Commit graph

51 commits

Author SHA1 Message Date
14ef9ff091 [#158] Separate init object reader from read itself
All checks were successful
/ Vulncheck (pull_request) Successful in 1m40s
/ Builds (1.19) (pull_request) Successful in 2m54s
/ Builds (1.20) (pull_request) Successful in 2m39s
/ DCO (pull_request) Successful in 3m48s
/ Lint (pull_request) Successful in 3m32s
/ Tests (1.19) (pull_request) Successful in 2m38s
/ Tests (1.20) (pull_request) Successful in 2m55s
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:32:05 +03:00
d531b13866 [#143] Add more context to some s3 errors
All checks were successful
/ DCO (pull_request) Successful in 1m18s
/ Vulncheck (pull_request) Successful in 1m50s
/ Lint (pull_request) Successful in 3m1s
/ Tests (1.19) (pull_request) Successful in 2m47s
/ Tests (1.20) (pull_request) Successful in 3m2s
/ Builds (1.19) (pull_request) Successful in 2m54s
/ Builds (1.20) (pull_request) Successful in 7m58s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:08:33 +03:00
23593eee3d [#111] Use request scope logger
Some checks failed
/ Lint (pull_request) Failing after 32s
/ Tests (1.19) (pull_request) Failing after 32s
/ Tests (1.20) (pull_request) Failing after 32s
/ Builds (1.19) (pull_request) Failing after 32s
/ Builds (1.20) (pull_request) Failing after 32s
/ Vulncheck (pull_request) Failing after 57s
/ DCO (pull_request) Failing after 1m0s
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-19 13:54:51 +03:00
4a6e3a19ce [#125] Handle negative Content-Length on put
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-02 14:16:19 +00:00
e487ee5b7d [#70] Add arrays of copies numbers for location constraints
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-05-03 13:48:26 +03:00
813aa2f173 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 17:38:08 +03:00
8e14ccad9f [#754] Remove duplicating CID/OID logging calls
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
(cherry picked from commit 8c178fc3c7)
2022-12-28 17:06:32 +03:00
af7fbd6f32 [#754] Add CID and OID to logs
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2022-12-28 17:06:32 +03:00
388482e230 [#2] Rename internals
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2022-12-21 11:17:45 +03:00
96dff367db [#1] Build S3 Gateway with FrostFS dependencies
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-15 12:43:52 +03:00
Denis Kirillov
094eb12578 [#726] Use client time on regular requests
Use `X-Amz-Date` header as `now` when
* compute expiration epoch
* set Timestamp for object and container
* forming locks
* send notifications

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-14 14:53:55 +03:00
Angira Kekteeva
da61fd8bcb [#728] Add error message to AbortMultipartUpload
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-10-25 19:04:11 +03:00
Denis Kirillov
821df3d648 [#669] Optimize getNodeVersion for tags operation
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-20 17:56:09 +03:00
Denis Kirillov
3d31c2ab4a [#713] Fix cache access
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-11 15:26:42 +03:00
Denis Kirillov
80beedf13e [#504] Use bktInfo in tree service to check owner
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-13 18:05:16 +03:00
Denis Kirillov
c3ad6d2faf [#637] Add header to override CopiesNumber
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-19 10:31:26 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-16 17:49:15 +03:00
Denis Kirillov
94a6a55919 [#595] Move decrypter to separate package
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-13 10:26:00 +03:00
Denis Kirillov
87c05eb514 [#595] Add SSE with customer key
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-13 10:26:00 +03:00
Alex Vanin
5219965105 [#498] Remove log string sanitizer
`zap` lib does all the job anyway.
Suppress CWE-117 from CodeQL.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-07-29 09:32:37 +03:00
Denis Kirillov
c88a9842db [#488] Sync using oid.ID
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
85c203e157 [#488] Sync using cid.ID
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
42a761c1f2 [#547] Cache ObjectInfo instead of Object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Angira Kekteeva
3f5afabd4a [#515] Fix nil reference when a part is missing
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
9dfc7e043f [#476] Save full parts info to multipart object
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
c8e8ba9f6a [#476] Fix parts info for GetObjectAttributes
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
9b1ccd39be [#475] Add extra attributes to partInfo
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
1a456eaa8b [#452] Remove unused
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
70957d75fd [#417] Complete multipart upload using tree service
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
6b2ddcadd0 [#417] Abort multipart upload using tree service
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
58f2bf44b1 [#417] List parts using tree service
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
bc0bdc7767 [#417] Upload part using tree service
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
e1b9a4432a [#417] List multipart uploads using tree service
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
13e01164d7 [#417] Create multipart upload using tree service
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-07-22 15:19:16 +03:00
Denis Kirillov
7ca519cb32 [#539] Add context to errors
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-27 02:23:19 +04:00
Angira Kekteeva
e904ed51c7 [#487] Optimize bucketInfo in initObjectPayloadReader
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-03 10:57:56 +03:00
Angira Kekteeva
4767eeed8c [#487] Remove attach of bearer token
When bucket owner is not an issuer of the bearer token

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-06-03 10:57:56 +03:00
Alex Vanin
12d9eb62cb [#489] Sanitize log records that may contain user input
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-06-02 19:42:31 +04:00
Denis Kirillov
2bca4755f9 [#441] Optimize put objects
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-05-31 14:48:58 +03:00
Leonard Lyubich
087d500c5f [#458] *: Refactor working with NeoFS identities
Pull latest changes from NeoFS SDK Go library. Decrease redundant and
unsafe usage of ID pointers. Use `EncodeToString` method in order to
calculate protocol strings.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 17:22:37 +04:00
Angira Kekteeva
7d69f9f74b [#432] Add get-object-attributes
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-05-13 11:42:43 +03:00
Denis Kirillov
e3c16a32dd [#409] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-26 12:51:52 +04:00
Elizaveta Chichindaeva
bf38007692 [#405] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-04-22 14:01:40 +03:00
Angira Kekteeva
f274747e83 [#400] Make multipart-upload parts system objects
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-04-19 12:51:04 +03:00
Denis Kirillov
f0914b8a43 [#377] Reuse BucketInfo in layer
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-29 02:43:28 +04:00
Leonard Lyubich
8fb3835250 [#346] api: Do not use io.Pipe in CompleteMultipartUpload
Replace `layer.objectWritePayload` method with `initObjectPayloadReader`
which returns `io.Reader` of the object payload. Copy payload data to
the parameterized `io.Writer` in `layer.GetObject`. Remove `io.Pipe`
from `CompleteMultipartUpload` implementation and build analogue of
`io.MultiReader` for the part list.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 00:14:30 +03:00
Denis Kirillov
a46726a545 [#331] Use UTC Last-Modified time
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-01-26 16:04:14 +03:00
Angira Kekteeva
8127d68128 [#186] Fix errors
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-30 11:32:59 +03:00