Commit graph

190 commits

Author SHA1 Message Date
Denis Kirillov
02f4524d67 [#368] Unify cache value deletion
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-16 10:59:42 +03:00
Denis Kirillov
e7b742df4d [#366] Add wait for container removed
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-04 10:41:15 +03:00
Leonard Lyubich
20b8e3d249 [#346] Upgrade NeoFS SDK Go to 2nd release candidate v1.0.0
Avoid using the deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
8fb3835250 [#346] api: Do not use io.Pipe in CompleteMultipartUpload
Replace `layer.objectWritePayload` method with `initObjectPayloadReader`
which returns `io.Reader` of the object payload. Copy payload data to
the parameterized `io.Writer` in `layer.GetObject`. Remove `io.Pipe`
from `CompleteMultipartUpload` implementation and build analogue of
`io.MultiReader` for the part list.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
eac4c4d849 [#346] api: Remove unused and no longer needed layer.Get method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
1517b5c638 [#346] neofs: Fix belated updateCRDT2PSetHeaders call
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Angira Kekteeva
4454821285 [#340] Add notification configuration handlers
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-18 16:59:42 +03:00
Angira Kekteeva
e0c6544567 [#340] Move notification controller to layer
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-18 16:59:42 +03:00
Denis Kirillov
f04815c32f [#354] Check container owner if bucket exists
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-02-15 17:08:46 +03:00
Denis Kirillov
c383f332d5 [#329] Use suitable tokens for container requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-31 11:05:35 +03:00
Denis Kirillov
8fc19b11db [#333] Update sdk
Update sdk to fix "invalid session token owner" error

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-28 12:42:40 +03:00
Angira Kekteeva
1502857dba [#313] Remove extra QueryUnescapes
Removed extra QueryUnescapes in search and objectPut

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-27 10:25:11 +03:00
Denis Kirillov
4f5b316211 [#319] Fix tests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:22:58 +03:00
Denis Kirillov
8985681493 [#319] Delete null versions in different modes
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:22:58 +03:00
Denis Kirillov
f5d365af1d [#319] Update CRDT headers
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:22:58 +03:00
Denis Kirillov
58df410111 [#319] Head and delete null versions
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:22:58 +03:00
Denis Kirillov
a46726a545 [#331] Use UTC Last-Modified time
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:04:14 +03:00
Denis Kirillov
962d136e73 [#308] Correct access denied status code
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-24 18:47:03 +03:00
Denis Kirillov
2b4ed8487b [#310] Support response headers overriding
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-21 16:02:06 +03:00
Denis Kirillov
c202893563 [#307] Filter system metadata
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-21 10:28:51 +03:00
Denis Kirillov
8872b6f196 [#285] Add resolving order
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-17 18:14:19 +03:00
Denis Kirillov
b9f77b3d96 [#301] Implement GetBucketLocation
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-13 20:24:21 +03:00
Denis Kirillov
9b14340816 [#299] Fix creating named containers
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-13 13:12:20 +03:00
Denis Kirillov
12c15504ef [#297] Update neofs-sdk-go and neo-go
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-12-30 13:00:23 +03:00
Angira Kekteeva
8127d68128 [#186] Fix errors
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Angira Kekteeva
8c80b16360 [#186] Add unittests for MultipartUpload
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Angira Kekteeva
d6f0ab8ea4 [#186] Fix versioning issues in UploadPart
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Angira Kekteeva
873622d4d5 [#186] Add MultipartUpload support
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Angira Kekteeva
284a560ea6 [#186] Refactor objectSearch, fix objectPut
Fixed panic in objectPut() if input reader is nil
Made objectSearch searching by any attribute

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Denis Kirillov
befe084900 [#219] Add container name resolving
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-24 13:24:39 +03:00
Denis Kirillov
d36dfe8c61 [#271] Update neo-sdk-go to the latest version
Refactoring invoking pool methods for anonymous requests.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-16 14:42:08 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-16 14:42:08 +03:00
Angira Kekteeva
91bed76010 [#274] Refactor system cache and cors
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-15 17:44:40 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-12 09:03:09 +03:00
Angira Kekteeva
ccf5db95a5 [#217] Refactor system objects
Moved into a separate file
getSystemObject renamed to headSystemObject, implemented getSystemObject
for system objects with payload
Refactor putSystemObjects
Moved systemCacheKey from data system_object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-12 09:03:09 +03:00
Angira Kekteeva
0460225869 [#272] Fix linter warning
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-01 12:25:54 +03:00
Denis Kirillov
3231ecab03 [#263] Add LWW to system objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-23 20:37:55 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 10:46:18 +03:00
Denis Kirillov
3c2e25f977 [#248] Support delete marker
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 09:44:39 +03:00
Denis Kirillov
458f9cf17b [#242] Fix versions sorting
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-07 17:06:22 +03:00
Angira Kekteeva
be08596c22 [#231] cache: Add invalidation of ListObjectsCache
In put operations ListObjectsCache remove entries which can contain
put object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [#236] api: Refactor caches: ObjectsList, Objects
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
239742f413 [#236] api: Create info.go for basic structs
Moved BucketInfo and ObjectInfo from layer and handler to api

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Denis Kirillov
5502fb97c3 [#196] Add tagging caching
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Denis Kirillov
987185b9e1 [#196] Add bucket tagging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Denis Kirillov
16da1aba64 [#196] Add DeleteObjectTagging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Denis Kirillov
4a2575fbf3 [#196] Add GetObjectTagging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Denis Kirillov
8b5ebe2ec2 [#196] Add PutObjectTagging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Angira Kekteeva
44a2f1b471 [#234] *: Update go.mod
Remove neofs-node dep
Update versions neofs reps

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-30 17:01:41 +03:00
Angira Kekteeva
dcac1aa3e3 [#230] layer: Change objects_list_cache's key
Remove access_key from objectsListCache key

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-26 12:54:58 +03:00
Denis Kirillov
4bb885d526 [#122] Refactor
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 14:49:21 +03:00
Denis Kirillov
11558124cd [#122] Add versioning cache
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
f6c51cc9ee [#122] Update listObjectVerions
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
9c058a70fd [#122] Add tests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
43185de52a [#122] Add list object versions
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
d5aef7566f [#122] Add delete versioned object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
3130784ee6 [#122] Add getting specific object version
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:13:12 +03:00
Denis Kirillov
f463522f34 [#122] Add versioning put object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:10:46 +03:00
Denis Kirillov
feb45d0633 [#122] Add replacing objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:10:46 +03:00
Denis Kirillov
c50a16a5e3 [#122] Add enabling versioning
New handlers:
PutBucketVersioning, GetBucketVersioning

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:10:46 +03:00
Denis Kirillov
efe11c271f [#49] Add basic ACL translation
Implement functions:
GetBucketACL, PutBucketACL, GetObjectACL,
PutObjectACL, GetBucketPolicy, PutBucketPolicy

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-18 17:20:17 +03:00
Angira Kekteeva
b1c6629b10 [#206] Make caches' options configurable
Now caches' options can be configured in .yanl file.
Export caches' variables (listobjects and objects).

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-18 10:16:27 +03:00
Angira Kekteeva
295be71f00 [#206] api: Add objects cache
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-18 10:16:27 +03:00
Angira Kekteeva
ae19eb4ad4 [#206] Refactor ListObjectsCache
Replaced conflicting names

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-18 10:16:27 +03:00
Denis Kirillov
d332096598 [#199] Refactor
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-11 13:17:37 +03:00
Angira Kekteeva
3cdcbf6e21 [#179] Add unittests for cache
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-09 13:52:11 +03:00
Angira Kekteeva
8a69c7cca0 [#179] api: Add cache for ListObjectsV1/V2
Refactored cache for ListObjects:
made cache common for all versions,
simplified: remove dependendence on token/startafter
add mitable cachelifetime.

Refactored listobjects

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-09 13:52:11 +03:00
Angira Kekteeva
0ceea95e11 [#112] Add cache to ListObjects and layer
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-28 17:05:11 +03:00
Angira Kekteeva
c24fe5cc21 [#112] Move getBoxData from handler to layer
And made it exported

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-28 16:27:06 +03:00
Angira Kekteeva
97a7d16f68 [#112] api: Refactor allObjectsParams struct
Replaced Bucket (means name) by BucketInfo struct

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-26 23:22:05 +03:00
Alex Vanin
f4cd1e4c38
Merge pull request #170 from KirillovDenis/feature/89-placement_policy
[#89] Add placement policy
2021-07-26 13:43:38 +03:00
Denis Kirillov
8aac61652f [#89] Move BoxData to CreateBucketParams
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-26 13:35:52 +03:00
Alex Vanin
3f7941fdf3
Merge pull request #174 from nspcc-dev/feature/159-optional_owner
[#159] Add fetch-owner param
2021-07-26 13:32:04 +03:00
Denis Kirillov
28974474f2 [#149] Refactoring
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-22 09:55:35 +03:00
Denis Kirillov
63652d213c [#149] Removed grpc errors
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-22 09:54:12 +03:00
Denis Kirillov
65a61a9f7d [#159] Add fetch-owner param
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-20 15:40:38 +03:00
Denis Kirillov
c05e2d6757 [#160] Fix head dir object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-20 12:36:41 +03:00
Angira Kekteeva
ee84062154 [#154] api: refactor ListObjectsV1 and V2
Separate ListObject for different versions.
Remove useless grouping of keys on V2.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-19 10:12:47 +03:00
Denis Kirillov
1fbd192bd7 [#89] Add placement policy
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-16 16:01:35 +03:00
Denis Kirillov
1ecf32c302 [#155] Fixed invalid max-keys handling
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-14 12:21:09 +03:00
Alex Vanin
7b1058a9bd Merge pull request #143 from KirillovDenis/bugfix/125-check_bucket_unique
[#125] Fixed bucket creation
2021-07-09 17:44:28 +03:00
Denis Kirillov
acc2cdd96c [#125] Added exact eacl checking
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-09 17:06:35 +03:00
Denis Kirillov
e78543adf3 [#125] Updated error handling
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-09 11:57:44 +03:00
Angira Kekteeva
d09a4e4e6c [#145] Fix util tests
Fixed contentType attribute difference in expected and actual results

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-09 00:36:19 +03:00
Denis Kirillov
e11b1b76ba [#125] Added eacl to newly created container
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-08 13:10:46 +03:00
Denis Kirillov
0aae8c595a [#125] Fixed bucket creation
Bucket should has unique name.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-08 12:22:02 +03:00
Angira Kekteeva
aac1ac2868 [#119] Add null versioning and handler
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-06 12:11:03 +03:00
Denis Kirillov
e145effd17 [#129] Fix root objects listing
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-04 23:25:53 +03:00
Roman Khimov
daed0978a6 Merge pull request #126 from KirillovDenis/feature/94-time_based_conditional
[#94] CopyObject/GetObject support conditional headers
2021-07-02 17:19:16 +03:00
Denis Kirillov
42384730a0 [#121] Fixed directory listing
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-01 14:41:20 +03:00
Denis Kirillov
ab8dd4201c [#94] CopyObject support conditional headers
Supported X-Amz-Copy-Source-If-Modified-Since and
X-Amz-Copy-Source-If-Unmodified-Since.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-01 10:45:55 +03:00
Denis Kirillov
2af1b16b59 [#93] Removed unnecessary
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-30 17:25:46 +03:00
Denis Kirillov
a6ec27b40d [#93] Object ETag support
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-30 12:55:31 +03:00
Denis Kirillov
65be38831c [#98] Supported listObjects delimiter
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-30 10:50:47 +03:00
Denis Kirillov
23dfa0c46e [#97] List object paging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-28 17:05:14 +03:00
Denis Kirillov
c4fef3d948 [#96] Support ranges for GetObject
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-25 12:18:52 +03:00
Angira Kekteeva
26d6fd95c7 [#47] authmate, layer: use sdk-go Container impl
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-23 23:36:39 +03:00