Commit graph

796 commits

Author SHA1 Message Date
Denis Kirillov
818176e7e1 [] Adopt aws v4signer
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-20 09:44:25 +03:00
Leonard Lyubich
f0749fd23e [] Upgrade NeoFS SDK Go with changed netmap package
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 08:28:59 +03:00
Angira Kekteeva
cfe7591cf7 [] Add putObjectACL notification
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-17 01:43:46 +04:00
Angira Kekteeva
dd0d21b690 [] Fix typo
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-17 01:43:46 +04:00
Angira Kekteeva
bd5fd041b7 [] Add object tagging notifications
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-17 01:43:46 +04:00
Denis Kirillov
d521af2065 [] Add presign URLs support
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-16 17:17:45 +03:00
Leonard Lyubich
880ffe7108 [] layer: Don't parameterize basic ACL of created containers
`CreateBucket` handler always creates containers with extended public
ACL, so there is no need to configure it in `NeoFS.CreateContainer`.

Make internal `NeoFS` implementation to create containers with
`eacl-public-read-write` basic ACL if corresponding parameter is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-16 10:46:50 +03:00
Angira Kekteeva
a8bff13801 [] Add acl to copy-object
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-07 17:36:12 +03:00
Denis Kirillov
de7281ac58 [] Unify log messages
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-07 17:20:26 +03:00
Denis Kirillov
300d4359d8 [] Handle cache cast failure
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-07 17:20:26 +03:00
Denis Kirillov
f00ca1b6c4 [] Drop layer/neofs package
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-07 13:59:45 +03:00
Alex Vanin
4ed939773b [] Use updated SDK method to get bearer issuer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-06-03 11:20:47 +03:00
Angira Kekteeva
dbfac29171 [] Fix tests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-03 10:57:56 +03:00
Angira Kekteeva
e904ed51c7 [] Optimize bucketInfo in initObjectPayloadReader
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-03 10:57:56 +03:00
Angira Kekteeva
4767eeed8c [] Remove attach of bearer token
When bucket owner is not an issuer of the bearer token

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-03 10:57:56 +03:00
Leonard Lyubich
4f43aad495 [] Upgrade SDK with latest bearer package API
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 09:38:49 +03:00
Denis Kirillov
ea252421f5 [] Add tests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-03 09:35:06 +03:00
Denis Kirillov
f282e877e2 [] Handle conditional headers
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-03 09:35:06 +03:00
Alex Vanin
ee0f3fb196 [] Avoid sensitive data logging
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-06-02 19:42:31 +04:00
Alex Vanin
12d9eb62cb [] Sanitize log records that may contain user input
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-06-02 19:42:31 +04:00
Denis Kirillov
933ef2bc71 [] Remove unused
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-05-31 14:48:58 +03:00
Denis Kirillov
2bca4755f9 [] Optimize put objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-05-31 14:48:58 +03:00
Denis Kirillov
faa3c65290 [] Fix None versioning status
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-05-30 14:44:21 +03:00
Alex Vanin
0e37242b65 [] Move DefaultLocationConstraint constant into api
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-30 14:41:32 +03:00
Alex Vanin
80d4d071d8 [] Restrict overriding default location constraint in authmate
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-30 14:41:32 +03:00
Alex Vanin
89ff89a32b [] Define default location constraint
Fixes `test_bucket_get_location` from s3-tests.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-30 14:41:32 +03:00
Leonard Lyubich
087d500c5f [] *: Refactor working with NeoFS identities
Pull latest changes from NeoFS SDK Go library. Decrease redundant and
unsafe usage of ID pointers. Use `EncodeToString` method in order to
calculate protocol strings.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 17:22:37 +04:00
Alex Vanin
80c6122f55 [] Do not pass nil error to logAndSendError
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-25 00:36:42 +04:00
Alex Vanin
385437aace [] Don't produce extra object.Head request at regular object upload
Hash can be calculated locally in S3 gateway.
Creation epoch used for versioning and will be
fetched during get and list requests. To avoid
conflicts, put method do not update cache anymore.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-20 10:56:36 +03:00
Alex Vanin
2575462daa [] Less cache misses for empty bucket setting
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-20 10:56:16 +03:00
Angira Kekteeva
7d69f9f74b [] Add get-object-attributes
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-05-13 11:42:43 +03:00
Alex Vanin
b778c2e072 [] Fix ETag value
ETag is a string with SHA256 of NeoFS object payload.
Hash is taken from object header by neofs-sdk-go getter.
Checksum type in neofs-sdk-go has changes `String()`
output from `<hash>` to `SHA256:<hash>` in latest update.

S3 gateway should not be relied on unstable string format
implementations and use raw value.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-12 21:19:59 +04:00
Alex Vanin
1c33f06bfe [] Update SDK
Includes:
- container removal fix
- new session token structure: authmate does not
  parse session context anymore, instead it is
  application defined flexible structure with
  container ID encoded in human-readable format

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-06 14:47:06 +04:00
Denis Kirillov
94caa2247e [] Refactor notifications
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-05-06 10:59:31 +03:00
Denis Kirillov
e3c16a32dd [] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-26 12:51:52 +04:00
Denis Kirillov
6e91074b50 [] Check errors using status
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-22 15:30:42 +04:00
Elizaveta Chichindaeva
bf38007692 [] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-04-22 14:01:40 +03:00
Angira Kekteeva
a0a04a73bd [] Add setEACL session token checks
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-20 12:42:33 +03:00
Angira Kekteeva
f274747e83 [] Make multipart-upload parts system objects
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-19 12:51:04 +03:00
Denis Kirillov
3c5c2f20d8 [] Use bearer token for system objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-13 16:44:32 +04:00
Denis Kirillov
7710de39ec [] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-08 17:12:52 +03:00
Leonard Lyubich
01c721ee53 [] cmd/s3-gw: Refactor logger
Do not import `logger` package from NeoFS SDK Go. Discard unusable
`name` and `version` configuration values from `app` section. Discard
all unusable onfiguration values from `logger` section except `level`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-06 17:31:01 +03:00
Angira Kekteeva
f3df5ff633 [] Fix grantee in ACL
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-06 15:30:17 +03:00
Angira Kekteeva
ed47bc1596 Fixed deletes for failed tests
test_multi_object_delete
test_multi_objectv2_delete

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-06 09:55:00 +03:00
Angira Kekteeva
e319f2422e [] Remove checks of bucket settings is not nil
Via 
Also fixed test

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:57:19 +03:00
Angira Kekteeva
02bcbe9754 [] Replace prepareEvent params by struct
By struct SendNotificationParams
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:57:19 +03:00
Angira Kekteeva
a718b92652 [] Add ObjectDelete notifications
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:57:19 +03:00
Angira Kekteeva
021f5d4dd0 [] Refactor delete objects
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:57:19 +03:00
Angira Kekteeva
b7aac223df [] Add ObjectCreated notifications
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:57:19 +03:00
Angira Kekteeva
67c1ba2c61 [] Add tests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:57:19 +03:00
Angira Kekteeva
371eb2feda [] Add events and sending of events
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:57:19 +03:00
Angira Kekteeva
40e7dbf768 [] Add test events and check of bucket notif conf
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:57:19 +03:00
Angira Kekteeva
2b6843f8fa [] Fix checks of bucket settings
In Put/CopyObject and PutBucketVersioning

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-01 14:22:10 +04:00
Denis Kirillov
c399cfbdda [] Refactor neofs related types
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-31 11:47:24 +03:00
Angira Kekteeva
83bb6fc020 [] Add check of Notificator interface value
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-03-30 10:29:28 +03:00
Denis Kirillov
f0914b8a43 [] Reuse BucketInfo in layer
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-29 02:43:28 +04:00
Denis Kirillov
46e4b28489 [] Add logger to notification controller
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
9d19acadcd [] Use exp epoch attribute instead of tick
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
cfcc8933e4 [] Add response lock headers
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
e0b8bc6cd6 [] Fix object deletion
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
4a67e4b311 [] Set tick attribute to lock objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
4c3c4b6bee [] Add Lock tick listening
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
32e83db064 [] Set Lock type to system lock objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [] Refactor TestNeoFS
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
5c530123ab [] Fix lint errors
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
6cf01bed14 [] Add tests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
7d6271be8a [] Implement PUT, GET locks to certain object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
8553158b81 [] Add handling lock headers for PUT and COPY
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
fe9eb9cedc [] Add PUT and GET default lock configuration
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
b96c3c5a33 [] Refactor
Using object settings to save bucket versioning

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
3046d80b37 [] Check object locking when disable versioning
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
e98c663bd6 [] Support enabling object locking for bucket
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-23 14:32:13 +04:00
Denis Kirillov
94406b08f3 [] Update NeoFS SDK to v1.0.0-rc.3
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-16 15:44:19 +03:00
Denis Kirillov
02f4524d67 [] Unify cache value deletion
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-16 10:59:42 +03:00
Denis Kirillov
e7b742df4d [] Add wait for container removed
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-04 10:41:15 +03:00
Leonard Lyubich
20b8e3d249 [] Upgrade NeoFS SDK Go to 2nd release candidate v1.0.0
Avoid using the deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
8fb3835250 [] api: Do not use io.Pipe in CompleteMultipartUpload
Replace `layer.objectWritePayload` method with `initObjectPayloadReader`
which returns `io.Reader` of the object payload. Copy payload data to
the parameterized `io.Writer` in `layer.GetObject`. Remove `io.Pipe`
from `CompleteMultipartUpload` implementation and build analogue of
`io.MultiReader` for the part list.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
eac4c4d849 [] api: Remove unused and no longer needed layer.Get method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
1517b5c638 [] neofs: Fix belated updateCRDT2PSetHeaders call
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [] *: Refactor communication with NeoFS at the protocol level
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [] Upgrade NeoFS SDK Go library
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Angira Kekteeva
4454821285 [] Add notification configuration handlers
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-18 16:59:42 +03:00
Angira Kekteeva
4cbce87eac [] Make nats tls and ca params optional
nats.Connect returned error when tls and ca files were not set, what
made these params required, but establishing of unsecured connection
for debug is acceptable.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-18 16:59:42 +03:00
Angira Kekteeva
e0c6544567 [] Move notification controller to layer
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-18 16:59:42 +03:00
Denis Kirillov
f04815c32f [] Check container owner if bucket exists
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-02-15 17:08:46 +03:00
Angira Kekteeva
3277293bb3 [] Add notifications controller
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-02-04 14:41:22 +03:00
Denis Kirillov
c383f332d5 [] Use suitable tokens for container requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-31 11:05:35 +03:00
Denis Kirillov
9be4bbbed4 [] Support Expires and Cache-Control headers
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-31 00:04:41 +03:00
Denis Kirillov
8fc19b11db [] Update sdk
Update sdk to fix "invalid session token owner" error

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-28 12:42:40 +03:00
Angira Kekteeva
1502857dba [] Remove extra QueryUnescapes
Removed extra QueryUnescapes in search and objectPut

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-27 10:25:11 +03:00
Denis Kirillov
4f5b316211 [] Fix tests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:22:58 +03:00
Denis Kirillov
8985681493 [] Delete null versions in different modes
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:22:58 +03:00
Denis Kirillov
f5d365af1d [] Update CRDT headers
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:22:58 +03:00
Denis Kirillov
58df410111 [] Head and delete null versions
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:22:58 +03:00
Denis Kirillov
a46726a545 [] Use UTC Last-Modified time
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:04:14 +03:00
Denis Kirillov
962d136e73 [] Correct access denied status code
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-24 18:47:03 +03:00
Denis Kirillov
2b4ed8487b [] Support response headers overriding
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-21 16:02:06 +03:00
Angira Kekteeva
bffc09167a [] Remove minio mentions in commits
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-21 10:44:54 +03:00
Angira Kekteeva
e5a256df0c [] Return badrequest err if couldnt parse body
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-21 10:44:54 +03:00
Angira Kekteeva
3307953a4c [] Remove MinIO related errors
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-21 10:44:54 +03:00
Denis Kirillov
c202893563 [] Filter system metadata
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-21 10:28:51 +03:00
Denis Kirillov
d532eb710f [] Add test to check empty policy
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-18 13:23:32 +03:00
Denis Kirillov
76042f6728 [] Handle empty post policy
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-18 13:23:32 +03:00
Denis Kirillov
e8e07147a8 [] Add x-amz-version-id header
Support version id in CompleteMultipartUpload method

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-17 18:18:17 +03:00
Denis Kirillov
8872b6f196 [] Add resolving order
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-17 18:14:19 +03:00
Denis Kirillov
097f745d3e [] Fix UnknownAPIRequest handling
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-17 17:29:06 +03:00
Denis Kirillov
b9f77b3d96 [] Implement GetBucketLocation
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-13 20:24:21 +03:00
Denis Kirillov
9b14340816 [] Fix creating named containers
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-13 13:12:20 +03:00
Denis Kirillov
12c15504ef [] Update neofs-sdk-go and neo-go
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-12-30 13:00:23 +03:00
Angira Kekteeva
8127d68128 [] Fix errors
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Angira Kekteeva
8c80b16360 [] Add unittests for MultipartUpload
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Angira Kekteeva
d6f0ab8ea4 [] Fix versioning issues in UploadPart
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Angira Kekteeva
873622d4d5 [] Add MultipartUpload support
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Angira Kekteeva
284a560ea6 [] Refactor objectSearch, fix objectPut
Fixed panic in objectPut() if input reader is nil
Made objectSearch searching by any attribute

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00
Denis Kirillov
befe084900 [] Add container name resolving
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-24 13:24:39 +03:00
Denis Kirillov
d36dfe8c61 [] Update neo-sdk-go to the latest version
Refactoring invoking pool methods for anonymous requests.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-16 14:42:08 +03:00
Denis Kirillov
ae87effb28 [] Add random key for no sign requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-16 14:42:08 +03:00
Angira Kekteeva
91bed76010 [] Refactor system cache and cors
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-15 17:44:40 +03:00
Angira Kekteeva
5b4b9df031 [] Add errors for cors
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-12 09:03:09 +03:00
Angira Kekteeva
f4afe41086 [] Add headers in response to simple requests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-12 09:03:09 +03:00
Angira Kekteeva
7d0bc1e992 [] Add CORS support
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-12 09:03:09 +03:00
Angira Kekteeva
ccf5db95a5 [] Refactor system objects
Moved into a separate file
getSystemObject renamed to headSystemObject, implemented getSystemObject
for system objects with payload
Refactor putSystemObjects
Moved systemCacheKey from data system_object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-12 09:03:09 +03:00
Angira Kekteeva
0460225869 [] Fix linter warning
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-10-01 12:25:54 +03:00
Denis Kirillov
3231ecab03 [] Add LWW to system objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-23 20:37:55 +03:00
Denis Kirillov
d616d9e2d9 [] Fix status codes
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-23 14:22:24 +03:00
Alex Vanin
16296490ee [] Adopt minio code distributed with AGPLv3 license in gateway components
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-20 10:38:28 +03:00
Alex Vanin
15023b9371 [] Add minio code distributed with AGPLv3 license
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-20 10:38:28 +03:00
Alex Vanin
5d11d8f556 [] Remove minio code distributed with Apache 2.0 license
NeoFS S3 Gate based on Minio RELEASE.2020-07-02T00-15-09Z
code which distributed with Apache 2.0 license. This commit
removes remaining minio code with that license.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-20 10:38:28 +03:00
Denis Kirillov
19b917e3b5 [] Caches refactoring
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 10:46:18 +03:00
Denis Kirillov
951eb6fda8 [] Add access box cache
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 10:46:18 +03:00
Denis Kirillov
3c2e25f977 [] Support delete marker
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-14 09:44:39 +03:00
Denis Kirillov
458f9cf17b [] Fix versions sorting
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-07 17:06:22 +03:00
Denis Kirillov
42ed6a16ea [] Add POST object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-07 11:10:48 +03:00
Angira Kekteeva
b2bf81cec7 [] cache: Add unittests for ListObjectsCache
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
d4a6a3ea2c [] cache: Update comments for ListObjectsCache
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
be08596c22 [] cache: Add invalidation of ListObjectsCache
In put operations ListObjectsCache remove entries which can contain
put object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
1ece42b23f [] cache: Refactor ListObjectsCache
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [] api: Refactor caches: ObjectsList, Objects
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Angira Kekteeva
239742f413 [] api: Create info.go for basic structs
Moved BucketInfo and ObjectInfo from layer and handler to api

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-09-07 11:08:48 +03:00
Denis Kirillov
a9be642eaf [] Add object acl versioning
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 12:38:23 +03:00
Denis Kirillov
5502fb97c3 [] Add tagging caching
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Denis Kirillov
987185b9e1 [] Add bucket tagging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Denis Kirillov
16da1aba64 [] Add DeleteObjectTagging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Denis Kirillov
4a2575fbf3 [] Add GetObjectTagging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Denis Kirillov
8b5ebe2ec2 [] Add PutObjectTagging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-31 10:46:42 +03:00
Angira Kekteeva
44a2f1b471 [] *: Update go.mod
Remove neofs-node dep
Update versions neofs reps

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-30 17:01:41 +03:00
Angira Kekteeva
dcac1aa3e3 [] layer: Change objects_list_cache's key
Remove access_key from objectsListCache key

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-26 12:54:58 +03:00
Kirillov Denis
04b7958ab3
Merge pull request from KirillovDenis/feature/122-poc_versioning
[] Add enabling versioning
2021-08-25 15:01:33 +03:00
Denis Kirillov
4bb885d526 [] Refactor
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 14:49:21 +03:00
Angira Kekteeva
2299db4e81 [] handler,s3-gw: Make policy configurable
Now default policy of placing containers can be set via config/env
variable.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-25 12:18:23 +03:00
Angira Kekteeva
6d4fe34f3d [] handler: Remove unused struct
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-25 12:18:23 +03:00
Denis Kirillov
11558124cd [] Add versioning cache
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
f6c51cc9ee [] Update listObjectVerions
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
9c058a70fd [] Add tests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
43185de52a [] Add list object versions
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
d5aef7566f [] Add delete versioned object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:15:00 +03:00
Denis Kirillov
3130784ee6 [] Add getting specific object version
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:13:12 +03:00
Denis Kirillov
f463522f34 [] Add versioning put object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:10:46 +03:00
Denis Kirillov
feb45d0633 [] Add replacing objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:10:46 +03:00
Denis Kirillov
c50a16a5e3 [] Add enabling versioning
New handlers:
PutBucketVersioning, GetBucketVersioning

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:10:46 +03:00
Denis Kirillov
d81a3d7b45 [] Add bucket owner check
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-25 09:07:38 +03:00
Alex Vanin
68e4e1bbc3
Merge pull request from KirillovDenis/feature/220-add_cid_to_head_bucket_response
[] Add cid to HeadBucket response
2021-08-24 18:10:13 +03:00
Alex Vanin
036662feab
Merge pull request from KirillovDenis/feature/221-unify_source_filenames
[] Unify source file naming
2021-08-24 12:44:23 +03:00
Denis Kirillov
84508a1706 [] Add cid to HeadBucket response
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-23 11:37:08 +03:00
Denis Kirillov
76fc241bc1 [] Add condition to update eacl
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-20 16:32:51 +03:00
Denis Kirillov
bf3d81f928 [] Unify source file naming
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-20 15:19:49 +03:00
Denis Kirillov
efe11c271f [] Add basic ACL translation
Implement functions:
GetBucketACL, PutBucketACL, GetObjectACL,
PutObjectACL, GetBucketPolicy, PutBucketPolicy

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-18 17:20:17 +03:00
Angira Kekteeva
b1c6629b10 [] Make caches' options configurable
Now caches' options can be configured in .yanl file.
Export caches' variables (listobjects and objects).

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-18 10:16:27 +03:00
Angira Kekteeva
376642565f [] Add unit tests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-18 10:16:27 +03:00
Angira Kekteeva
295be71f00 [] api: Add objects cache
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-18 10:16:27 +03:00
Angira Kekteeva
ae19eb4ad4 [] Refactor ListObjectsCache
Replaced conflicting names

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-18 10:16:27 +03:00
Denis Kirillov
4ca56d3d6e [] Add fine-grained handle auth header
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-11 13:17:37 +03:00
Denis Kirillov
d332096598 [] Refactor
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-11 13:17:37 +03:00
Denis Kirillov
f3a6636efd [] Improve copy object compatibility
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-11 12:37:26 +03:00
Denis Kirillov
7eb9713a67 [] Add bucket name checking
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-11 12:04:00 +03:00
Kirillov Denis
b555a1b1d8
Merge pull request from masterSplinter01/feature/179-add-cache-to-lov1
Add cache to ListObjectsV1
2021-08-09 14:20:20 +03:00
Angira Kekteeva
3cdcbf6e21 [] Add unittests for cache
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-09 13:52:11 +03:00
Angira Kekteeva
8a69c7cca0 [] api: Add cache for ListObjectsV1/V2
Refactored cache for ListObjects:
made cache common for all versions,
simplified: remove dependendence on token/startafter
add mitable cachelifetime.

Refactored listobjects

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-09 13:52:11 +03:00
Denis Kirillov
a89add8ff3 [] Fix object metadata
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-06 15:06:40 +03:00
Denis Kirillov
e5b1dae750 [] Fix range header handling
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-06 13:14:50 +03:00
Denis Kirillov
45110b4c5d [] Fix typo
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-05 12:29:31 +03:00
Denis Kirillov
f4c29cd300 [] Unify error handling
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-08-05 12:18:52 +03:00
Angira Kekteeva
eaec271283 [] api: Add unittests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-03 12:40:28 +03:00
Angira Kekteeva
88aaad361a [] api: Add continuation-token check
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-08-03 12:40:02 +03:00
Angira Kekteeva
7291b81040 [] api: Format object_list.go
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-29 17:31:48 +03:00
Angira Kekteeva
afff616675 [] api: Remove extra parses of url requests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-29 17:18:51 +03:00
Angira Kekteeva
0ceea95e11 [] Add cache to ListObjects and layer
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-28 17:05:11 +03:00
Angira Kekteeva
c24fe5cc21 [] Move getBoxData from handler to layer
And made it exported

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-28 16:27:06 +03:00
Angira Kekteeva
97a7d16f68 [] api: Refactor allObjectsParams struct
Replaced Bucket (means name) by BucketInfo struct

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-26 23:22:05 +03:00
Denis Kirillov
2479650739 [] Correct NotFound status
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-26 14:03:26 +03:00
Alex Vanin
f4cd1e4c38
Merge pull request from KirillovDenis/feature/89-placement_policy
[] Add placement policy
2021-07-26 13:43:38 +03:00
Denis Kirillov
8aac61652f [] Move BoxData to CreateBucketParams
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-26 13:35:52 +03:00
Alex Vanin
3f7941fdf3
Merge pull request from nspcc-dev/feature/159-optional_owner
[] Add fetch-owner param
2021-07-26 13:32:04 +03:00
Denis Kirillov
c7cf5afd2f [] Add benchmarks
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-22 09:55:35 +03:00
Denis Kirillov
60bc0037fd [] Remove unused
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-22 09:55:35 +03:00
Denis Kirillov
28974474f2 [] Refactoring
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-22 09:55:35 +03:00
Denis Kirillov
63652d213c [] Removed grpc errors
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-22 09:54:12 +03:00
Denis Kirillov
b695e6a3b4 [] Refactor error logging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-22 09:45:24 +03:00
Denis Kirillov
65a61a9f7d [] Add fetch-owner param
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-20 15:40:38 +03:00
Denis Kirillov
c05e2d6757 [] Fix head dir object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-20 12:36:41 +03:00
Angira Kekteeva
b1cda2a714 [] api: refactor EncodeV1 and EncodeV2
Move common parts of Encode to separate functions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-19 16:38:51 +03:00
Angira Kekteeva
ee84062154 [] api: refactor ListObjectsV1 and V2
Separate ListObject for different versions.
Remove useless grouping of keys on V2.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-19 10:12:47 +03:00
Denis Kirillov
1fbd192bd7 [] Add placement policy
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-16 16:01:35 +03:00
Angira Kekteeva
b7ebe4127f [] Change delimiter in acesskeyID
Replaced delimiter '_' by '0'

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-16 14:17:01 +03:00
Alex Vanin
22faaadc32 Merge pull request from nspcc-dev/feature/158-s3_tests_conditional_headers
[] Handled s3 errors on conditional headers
2021-07-16 09:39:17 +03:00
Denis Kirillov
352d5345fc [] Handled s3 errors on conditional headers
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-16 09:30:08 +03:00
Alex Vanin
3a7876e292 Merge pull request from KirillovDenis/feature/155-s3_ceph_listObject_compatibility
[] Improved s3 listObjects compatibility
2021-07-15 18:48:26 +03:00
Denis Kirillov
f2eeed0b85 [] Fix error handling
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-14 19:06:58 +03:00
Denis Kirillov
1ecf32c302 [] Fixed invalid max-keys handling
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-14 12:21:09 +03:00
Denis Kirillov
9f57756351 [] Added s3 url encoder
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-14 11:34:49 +03:00
Denis Kirillov
288f6edce8 [] Fixed default maxKey value
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-13 17:57:48 +03:00
Denis Kirillov
5fb4c4fad6 [] Fixed empty delimiter response
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-13 17:46:32 +03:00
Denis Kirillov
b004996d51 [] Fix KeyCount
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-13 17:45:38 +03:00
Denis Kirillov
e1c2cee365 [] Removed redundant GetBucketInfo call
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-13 14:22:12 +03:00
Alex Vanin
7b1058a9bd Merge pull request from KirillovDenis/bugfix/125-check_bucket_unique
[] Fixed bucket creation
2021-07-09 17:44:28 +03:00
Alex Vanin
1f656aac23 Merge pull request from masterSplinter01/bugfix/131-add-metadata
Fix metadata
2021-07-09 17:20:59 +03:00
Denis Kirillov
acc2cdd96c [] Added exact eacl checking
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-09 17:06:35 +03:00
Angira Kekteeva
d5dfc0006a [] Remove metadata from Object in ListObjects
Because ListObjects v1 and v2 shouldn't show metadata. Also aws cli
doesn't show metadata even we return it.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-09 13:44:24 +03:00
Angira Kekteeva
da9a6e4b33 [] Fix metadata in HeadObject and GetObject
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-09 13:44:24 +03:00
Angira Kekteeva
e8bf46a2fa [] Add meta to PutObject
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-09 13:44:24 +03:00
Denis Kirillov
e78543adf3 [] Updated error handling
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-09 11:57:44 +03:00
Angira Kekteeva
d09a4e4e6c [] Fix util tests
Fixed contentType attribute difference in expected and actual results

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-09 00:36:19 +03:00
Denis Kirillov
e11b1b76ba [] Added eacl to newly created container
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-08 13:10:46 +03:00
Denis Kirillov
6ba5167f5d [] Fixed acl rule
Container must be public (basic acl) to enable bearer token.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-08 12:22:02 +03:00
Denis Kirillov
0aae8c595a [] Fixed bucket creation
Bucket should has unique name.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-08 12:22:02 +03:00
Alex Vanin
f873be8019 Merge pull request from masterSplinter01/bugfix/135-max-keys-check
[] Add max-keys check to ListObjects
2021-07-06 15:03:43 +03:00
Alex Vanin
2956f34fb7 Merge pull request from KirillovDenis/bugfix/133-wrong_time_format
[] Fix response time format
2021-07-06 15:03:15 +03:00
Denis Kirillov
bdc7f9acba [] Fix response time format
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-06 14:07:09 +03:00
Angira Kekteeva
fd41c325f7 [] Add max-keys check to ListObjects
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-06 12:23:46 +03:00
Angira Kekteeva
aac1ac2868 [] Add null versioning and handler
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-07-06 12:11:03 +03:00
Angira Kekteeva
cc0e2e7636 Merge pull request from KirillovDenis/bugfix/129-fix_listing
[] Fix root objects listing
2021-07-05 12:20:26 +03:00
Denis Kirillov
e145effd17 [] Fix root objects listing
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-04 23:25:53 +03:00
Denis Kirillov
95476b1c9c [] Supported copy match headers
Supported x-amz-copy-source-if-match
and x-amz-copy-source-if-none-match.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-02 19:21:53 +03:00
Denis Kirillov
568d7ac810 [] Supported get match headers
Supported If-Match and If-None-Match.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-02 19:05:43 +03:00
Roman Khimov
daed0978a6 Merge pull request from KirillovDenis/feature/94-time_based_conditional
[] CopyObject/GetObject support conditional headers
2021-07-02 17:19:16 +03:00
Denis Kirillov
42384730a0 [] Fixed directory listing
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-01 14:41:20 +03:00
Denis Kirillov
116ffbb438 [] Fix time format
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-01 14:25:16 +03:00
Denis Kirillov
c4631e5806 [] Refactoring
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-01 14:24:00 +03:00
Denis Kirillov
ab8dd4201c [] CopyObject support conditional headers
Supported X-Amz-Copy-Source-If-Modified-Since and
X-Amz-Copy-Source-If-Unmodified-Since.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-01 10:45:55 +03:00
Denis Kirillov
da4eca5da5 [] GetObject support conditional headers
Supported If-Modified-Since and If-Unmodified-Since.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-01 08:39:19 +03:00
Denis Kirillov
2af1b16b59 [] Removed unnecessary
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-30 17:25:46 +03:00
Denis Kirillov
a6ec27b40d [] Object ETag support
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-30 12:55:31 +03:00
Denis Kirillov
b565045649 [] Fixed order headers set
Writing headers had no effect early.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-30 12:55:31 +03:00
Denis Kirillov
65be38831c [] Supported listObjects delimiter
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-30 10:50:47 +03:00
Denis Kirillov
23dfa0c46e [] List object paging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-28 17:05:14 +03:00
Roman Khimov
ce65a47d1b Merge pull request from KirillovDenis/feature/104-drop_neofs_crypto
[] Support NEP-6 for authmate
2021-06-26 16:40:44 +03:00
Denis Kirillov
c4fef3d948 [] Support ranges for GetObject
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-25 12:18:52 +03:00
Denis Kirillov
52c63d4c44 [] Support NEP-6 for authmate
Drop neofs-crypto.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-25 12:16:24 +03:00
Angira Kekteeva
26d6fd95c7 [] authmate, layer: use sdk-go Container impl
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-23 23:36:39 +03:00
Angira Kekteeva
3aa9aceda5 [] handler, layer: Add DeleteBucket
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-23 23:25:00 +03:00
Angira Kekteeva
c12abf6243 [] handler, layer: Add CreateBucket
Renamed PutBucket to CreateBucket

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-23 23:21:15 +03:00
Angira Kekteeva
95f75ec880 [] Add session token to context
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-21 13:56:26 +03:00
Denis Kirillov
d1594b586e [] Use multiple bearer tokens
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-18 11:19:07 +03:00
Denis Kirillov
b86d20265d [] Using secp256r1 instead of curve25519
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-17 16:36:23 +03:00
Angira Kekteeva
0814bf3d01 Merge pull request from roman-khimov/lint
Linter fixes
2021-06-16 23:01:29 +03:00
Roman Khimov
996b2d8667 auth: add missing comment
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-06-15 18:50:10 +03:00
Denis Kirillov
6a135a6bb4 [] Using bearer token
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-15 17:16:26 +03:00
Denis Kirillov
fba29a9a66 [] Using client.Object from sdk
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-15 17:06:19 +03:00
Roman Khimov
ee078eabcc Merge pull request from masterSplinter01/feature/48-protobuf-format-authbox
Protobuf format of accessbox
2021-06-15 16:57:22 +03:00
Angira Kekteeva
fe2d507121 [] creds,authmate:Replace old accessbox by new
Removed encoder, decoder wraps.
Made changes in api, authmate and creds via new accessbox.
Updated bearer_token_tests via new accessbox.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-15 14:54:11 +03:00
Denis Kirillov
47fc10a869 [] Using containerList falls back
Looking container up using the owner list if its name is not cid

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-13 14:32:13 +03:00
Denis Kirillov
4df647baac [] Added NoAuthorizationHeader error
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-11 19:29:55 +03:00
Denis Kirillov
8185b71462 [] Allow no sign requests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-11 14:52:03 +03:00
Angira Kekteeva
b3b190adc0 [] *: Replace deprecated
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-04 16:01:42 +03:00
Angira Kekteeva
1a818bac33 [] authmate, auth: Fix access key id
Replaced forbidden symbol '/' in access key id by '_'
SecretKeyAddress and SecretKeyID are different things now
Renamed param of authmate from secretAddressFlag to accessKeyIDFlag,
that is more correct, imo.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-03 22:24:02 +03:00
Roman Khimov
d15687f37c *: drop creds/neofs
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-28 23:48:39 +03:00
Roman Khimov
35ffc1fbc1 *: use sdk-go instead of http-gw
Now that the packages have moved there we can do it.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-28 14:05:46 +03:00
Roman Khimov
1c277bd97b api: fix golint suggestion for proper context usage
api/user-auth.go:27:5  golint  should not use basic type string as key in context.WithValue

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-26 21:25:37 +03:00
Roman Khimov
184c26551a *: fix comments for golint
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-26 21:23:48 +03:00
Roman Khimov
d19ce03072 *: drop old sdk dependecies, bump neofs-api-go version
I'm not sure it works, but it's enough code-wise for now. We're reusing some
http-gw components here that are to be moved into sdk-go in future.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-26 21:01:46 +03:00
Roman Khimov
dbe65ae602 creds: move credential management into s3 gate
Mostly taken from old SDK (abe47687cd11266f946cad57f07572cc10c67226), but
error handling adapted to eliminate pkg/errors and internal packages.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-25 23:00:19 +03:00
Roman Khimov
ce7c8932d4 authmate: drop creds/s3 dependency
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-25 19:52:51 +03:00
Roman Khimov
92c8cce933 misc: move to internal/version, use for all binaries
No libmisc, please.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-20 16:26:41 +03:00
Roman Khimov
ef8684c11d s3-gw: drop build time setting
It's not properly set anyway and it's a bad practice for build
reproducibility.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-20 16:26:20 +03:00
Angira Kekteeva
9d496d70a7 [] *: Remove external pkg/errors dependency
Replaced functions from pkg/errors by functions from errors, fixed not fully correct comment

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-20 15:50:14 +03:00
Angira Kekteeva
a399590204 Replace s3-gate by s3-gw
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-18 14:12:24 +03:00
Roman Khimov
5eb863dc22 *: fix golint warnings about comments to exported things. 2021-05-13 23:25:31 +03:00
Roman Khimov
70a70bfa2c *: fix all godot errors 2021-05-13 22:25:32 +03:00
Roman Khimov
a0ecb8ff52 handler: suppress exhaustive linter 2021-05-13 22:15:21 +03:00
Roman Khimov
b851889934 *: fix whitespace errors 2021-05-13 22:13:09 +03:00
Evgeniy Kulikov
8467a07c9b Update dependencies and fixes
- Closes 
- Cleanup code
- Make code more readable
- Update dependencies
	- github.com/aws/aws-sdk-go v1.37.1
	- github.com/google/uuid v1.2.0
	- github.com/nspcc-dev/cdn-sdk v0.3.3
	- github.com/nspcc-dev/neofs-api-go v1.22.2

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-01 19:59:42 +03:00
Evgeniy Kulikov
2a93a216f8 Migrate to SDK 0.3.0 and fixes
- fix displaying list objects
- simplify `ListObjects`
- simplify `GetObjectHandler`
- simplify `HeadObjectHandler`
- add method for `GetBucketVersioningHandler`
- add method for `ListMultipartUploadsHandler`
- improvements for `HeadObjectHandler`, to display folders meta
- update dependencies
  - github.com/aws/aws-sdk-go v1.36.26
  - github.com/google/uuid v1.1.4
  - github.com/gorilla/mux v1.8.0
  - github.com/nspcc-dev/cdn-sdk v0.3.0
  - github.com/nspcc-dev/neofs-api-go v1.22.0
  - github.com/prometheus/client_golang v1.9.0
  - github.com/stretchr/testify v1.7.0
  - google.golang.org/grpc v1.35.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-01-14 20:39:48 +03:00
Evgeniy Kulikov
8ff289ea4f Simplify OffsetWriter
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-12-10 18:15:43 +03:00
Evgeniy Kulikov
8168181658 Remove bufio writer and double creation of OffsetWriter
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-12-10 18:15:18 +03:00
Evgeniy Kulikov
776eed76fb Add writer detector
WriterDetector allows to fetch ContentType and write status response, when streaming content

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-12-10 18:14:32 +03:00
Evgeniy Kulikov
ef0e179b33 Fixes over bucket router
should work over {bucket}.{domain} and /{bucket}

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-12-10 18:13:28 +03:00
Evgeniy Kulikov
5752f2f89f Add method to fetch SecretAccessKey from BearerToken
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-27 16:33:56 +03:00
Evgeniy Kulikov
f4263d50a7 Refactoring api/layer
- don't require token.BearerToken
- simplify method layer.objectPut
- add method to check that object exists
- add method to fetch Owner (from context or sdk client)

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-27 15:36:15 +03:00
Evgeniy Kulikov
31d3e55489 Add content detector
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-27 15:33:31 +03:00
Evgeniy Kulikov
1868034723 Refactoring api/handler
- don't require token.BearerToken
- don't call `GetObjectInfo`

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-27 15:32:52 +03:00
Evgeniy Kulikov
58b877b97c Refactoring auth.Center
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-27 15:31:39 +03:00
Evgeniy Kulikov
697d318a6b WriteHeader should be called once
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-27 15:28:27 +03:00
Evgeniy Kulikov
1cd636a24b Refactoring api/auth package
Migrate to cred/bearer instead of CredentialsClient

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-24 19:31:57 +03:00
Evgeniy Kulikov
0284bf136b Migrate to Credentials client
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-24 12:17:36 +03:00
Evgeniy Kulikov
29cb755d49 Fixes of usage auth package
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-24 10:02:10 +03:00
Evgeniy Kulikov
0ee3a5f9ba Migrate api/layer to SDK
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-24 10:01:38 +03:00
Evgeniy Kulikov
e3b1e8f369 Fixes of usage auth package
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-24 10:00:49 +03:00
Evgeniy Kulikov
1fecf6a7ac Remove pool package
It will be replaced with sdk/pool

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-24 09:59:32 +03:00
Evgeniy Kulikov
b9972042fc Refactoring auth package and move into API
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-24 09:59:01 +03:00
Evgeniy Kulikov
54414c2bfb Fix NPE when response isn't http.Flusher
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-19 16:01:52 +03:00
Evgeniy Kulikov
256850b8fe [] Refactoring and make fixes
closes 
closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-24 16:09:22 +03:00
Evgeniy Kulikov
4d605d1113 [] Refactoring and make fixes
closes 
closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-23 03:21:14 +03:00
Evgeniy Kulikov
fbd4a83602 [] Refactoring over api/layer
closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-22 03:19:16 +03:00
Evgeniy Kulikov
b72a957341 [] Migrate handler to NeoFS API v2
Naive migration, without any tries to start application

closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-19 04:05:08 +03:00
Evgeniy Kulikov
7deeb68d47 [] Migrate layer to NeoFS API v2
Naive migration, without any tries to start application

closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-19 04:04:37 +03:00
Evgeniy Kulikov
86e07c8882 [] Fix linter warnings
closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-13 12:43:24 +03:00
Evgeniy Kulikov
b346a77933 [] Migrate connection pool to NeoFS API v2
closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-13 12:32:23 +03:00
Evgeniy Kulikov
8c93dbf867 [] Implement ListObjectsV2
ListObjectsV1 and ListObjectsV2 now simplified and return different results

closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:49:00 +03:00
Evgeniy Kulikov
079e7a9827 [] Implement HeadBucket
closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:45:48 +03:00
Evgeniy Kulikov
fd2c6e372b [] Implement DeleteMultipleObjects
closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:45:15 +03:00
Evgeniy Kulikov
cfe2811efc Add owner field for ListObjects method
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:43:53 +03:00
Evgeniy Kulikov
3244f54022 Change returned error for GetBucketInfo method
method should returns gRPC status errors

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:43:06 +03:00
Evgeniy Kulikov
8eff7d4dd3 Refactoring DeleteObject/DeleteObjects methods
- methods should return api.DeleteErrors
- DeleteObjects should return only slice of errors
- don't add nil errors

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:42:27 +03:00
Evgeniy Kulikov
960aa02bd3 Refactoring DeleteObject/DeleteObjects methods
- methods should return api.DeleteErrors
- DeleteObjects should return only slice of errors
- don't add nil errors

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:41:58 +03:00
Evgeniy Kulikov
5eae2aac78 Change NotFound error for objectFindID
method should return gRPC status error

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:39:12 +03:00
Evgeniy Kulikov
5258342881 Add Owner field into layer.ObjectInfo
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:37:39 +03:00
Evgeniy Kulikov
d2d54cbdaa Refactoring getAPIErrorResponse method
should receive base error type instead of custom errors

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:36:53 +03:00
Evgeniy Kulikov
0b533e189e Add custom http headers constants
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:35:32 +03:00
Evgeniy Kulikov
f3b202700e Refactoring WriteErrorResponse method
- should receive error instead custom type
- change visibility of WriteResponse method

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:34:50 +03:00
Evgeniy Kulikov
cd2cca8fb3 Change MimeTypes constants visibility
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-22 05:32:48 +03:00
Evgeniy Kulikov
0bea843d90 Fix DeleteObject handler
DeleteObject should remove all objects with passed nice name

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 22:41:52 +03:00
Evgeniy Kulikov
34f6eb7d55 Add note about multiple delete of objects
Could not find how to use it.
I propose open issue and describe how to use it and then implement method.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 22:35:39 +03:00
Evgeniy Kulikov
21591ebe65 Add method to find all ID's for the nice names
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 22:33:59 +03:00
Evgeniy Kulikov
5bb2c34052 Cleanup unimplemented handlers
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:37:06 +03:00
Evgeniy Kulikov
1d98c4ecc2 Implement GetObject handler
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:36:46 +03:00
Evgeniy Kulikov
bfc4b8786f Implement HeadObject handler
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:36:34 +03:00
Evgeniy Kulikov
9493f3d281 Implement PutObject handler
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:36:17 +03:00
Evgeniy Kulikov
9746978b1d Implement DeleteObject handler
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:36:00 +03:00
Evgeniy Kulikov
9ab0571d29 Implement CopyObject handler
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:35:51 +03:00
Evgeniy Kulikov
c6d4b6d731 Implement MarshalXML for StringMap type
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:34:47 +03:00
Evgeniy Kulikov
fb13d1fcb3 Move ListObjects and ListBuckets in same file
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:33:43 +03:00
Evgeniy Kulikov
edd44ea981 Check that aws filename wasn't set
we should check that filename header wasn't set, otherwise we should not change custom filename

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:32:42 +03:00
Evgeniy Kulikov
00a17d8316 CopyObject should contains custom headers
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:31:04 +03:00
Evgeniy Kulikov
fb00af23fb Fixed possible NPE in PutObject
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:30:18 +03:00
Evgeniy Kulikov
447a255d18 Add func to debug requests
- logging middleware
- response writer with status code

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:28:16 +03:00
Evgeniy Kulikov
5df041f7d7 Add func to write headers only
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-20 02:27:13 +03:00
Evgeniy Kulikov
cea8f9d4ca Add bucket info handler
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-19 14:53:17 +03:00
Evgeniy Kulikov
92b039fa5e Add unsupported handlers
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-19 14:52:44 +03:00
Evgeniy Kulikov
b14e2d54c8 Prepare list-objects handler
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-11 17:37:20 +03:00
Evgeniy Kulikov
ccd481e4e2 Prepare list-buckets handler
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-11 17:36:52 +03:00
Evgeniy Kulikov
d9f691a59a Refactoring API handlers
- add responses for list-objects and list-buckets
- set `implement me` errors more useful

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-11 17:35:54 +03:00
Evgeniy Kulikov
b2a2b5478b Refactoring API
- api.Error should implement error interface
- add method that encodes the response into ResponseWriter

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-11 17:34:06 +03:00
Evgeniy Kulikov
64429d736d Refactoring API layer
- logging RequestID
- should return error, when headers already received

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-11 14:32:04 +03:00
Evgeniy Kulikov
59e26d45fd Refactoring API router
- method to fetch request id
- middleware to set request id and info
- fixes for getAPIErrorResponse (fix possible NPE)

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-11 14:30:02 +03:00
Evgeniy Kulikov
26f1df2721 Refactoring request info
- simplify
- more usable

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-11 14:27:31 +03:00
Evgeniy Kulikov
0161d2fbd3 [] Rename go module name according to NSPCC standards
- refactoring s3 gate structure
- cleanup unused code
- rename go module to `github.com/nspcc-dev/neofs-s3-gate`

closes 

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-08-06 15:02:13 +03:00