Commit graph

53 commits

Author SHA1 Message Date
Denis Kirillov
b2db6300c4 [#222] Fix zip streaming
Skip objects with invalid FilePath

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-08 14:34:33 +03:00
Denis Kirillov
c258db6ab3 [#218] Unify system attribute format for GET/HEAD
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-28 16:14:49 +03:00
Denis Kirillov
b567a08a85 [#206] Refactor neofs 'not found' errors
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-30 10:06:05 +03:00
Denis Kirillov
3fa0e3caef [#200] Update docs
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-16 17:08:20 +03:00
Denis Kirillov
82eba97505 [#200] Unregister metrics on shutdown to fix test
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-16 17:08:20 +03:00
Denis Kirillov
ce84dc7068 [#200] Reload upload/download settings on SIGHUP
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-16 17:08:20 +03:00
Denis Kirillov
9ebcb6b964 [#188] Update SDK to avoid panic on go1.19
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-24 17:57:40 +03:00
Denis Kirillov
0fb75a1cde [#178] Update SDK to v1.0.0-rc.5
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-25 14:27:24 +03:00
Denis Kirillov
85ee0c44a2 [#163] Fix zip streaming
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-20 09:42:06 +03:00
Denis Kirillov
a42606742a [#142] Support resolving container nicename
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-28 17:18:13 +03:00
Denis Kirillov
2b780c1772 [#142] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-28 17:18:13 +03:00
Elizaveta Chichindaeva
11283c1c79 [#153] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-04-22 18:05:51 +03:00
Alex Vanin
1e3df95eed [#145] Use application context in NeoFS API requests
It is meaningless to use RequestCtx as a context.Context
for NeoFS operation, because context won't be closed
until application shutdown. Moreover, it also triggers
data race detection, because server's done channel, which
is accessible for reading from RequestCtx, is set to `nil`.

Using application context doesn't change gateway behavior,
but it suppresses data race trigger at shutdown. It also
allows possibility to set configurable timeouts for NeoFS
networking if we will ever need them.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-22 15:33:11 +04:00
Denis Kirillov
4f2b21a14b [#144] Add more verbose error output
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-22 15:14:52 +04:00
Denis Kirillov
1c2fec8182 [#147] Download zip by FilePath attribute
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-19 20:10:00 +04:00
Denis Kirillov
baf425453d [#139] fix lint
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-08 00:23:12 +04:00
Denis Kirillov
5080b43a04 [#139] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-08 00:23:12 +04:00
Denis Kirillov
6d362894ad [#132] Update NeoFS SDK to v1.0.0-rc.3
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-16 17:15:21 +03:00
Leonard Lyubich
f6ab94027c [#128] downloader: Avoid var naming collision
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 13:43:34 +03:00
Leonard Lyubich
271451dc32 [#128] downloader: Simplify detecting the Content-Type from payload
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 13:43:34 +03:00
Leonard Lyubich
c482bbd25a [#126] downloader: Replace Read with Iterate on ObjectListReader
Make `Downloader.DownloadZipped` to call `Iterate` method instead of
`Read` one during processing the `ObjectListReader`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-03 18:25:53 +03:00
Leonard Lyubich
2b7e4a36fb [#126] Upgrade NeoFS SDK Go library
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-03 18:25:53 +03:00
Denis Kirillov
39c29c6d6a [#125] Set object ids in a separate function
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-02-15 12:21:42 +03:00
Denis Kirillov
2764fabf04 [#125] Use the same HEAD, GET headers formation
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-02-15 12:21:42 +03:00
Angira Kekteeva
9d085740e0 [#123] Add url-encoded queries for prefix values
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-27 16:51:11 +03:00
Angira Kekteeva
c11b2332f9 [#123] Add url-encoded queries for attributes
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-01-27 16:51:11 +03:00
Angira Kekteeva
1bb6c5f384 [#111] Fix X- headers
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-12-09 16:12:13 +03:00
Angira Kekteeva
d6dd244756 [#111] Fix expiration epoch headers
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-12-09 16:12:13 +03:00
Angira Kekteeva
dbbc9e05cf [#111] Move attributes to a separate file
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-12-09 16:12:13 +03:00
Alex Vanin
79765fb7cc [#106] Update neofs-sdk-go to the latest version
Includes:
 - fix for unavailable endpoints of client pool,
 - session token cache with request retry in case
   of missing session token,
 - migrated neofs client.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-15 14:46:50 +03:00
Denis Kirillov
6265fcb26a [#105] Add newline to errors
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-12 15:32:18 +03:00
Denis Kirillov
9b92c0146c [#96] Add zip compression
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-01 11:27:22 +03:00
Denis Kirillov
0b364504a7 [#92] Support zip download
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-10-22 18:36:09 +03:00
Denis Kirillov
d2b0532929 [#85] Add integration tests
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-09-07 13:14:46 +03:00
Denis Kirillov
03a3f986ff [#30] Fix not found responses
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-20 13:40:39 +03:00
Denis Kirillov
87918483c6 [#76] Refactor HeadObject logic
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-15 10:57:22 +03:00
Denis Kirillov
d856fdf4c4 [#76] Added HEAD methods
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-14 18:56:27 +03:00
Roman Khimov
dee182a0c1 downloader: fix Last-Modified header
It should never ever be sent in local time, instead of
  Last-Modified: Tue, 29 Jun 2021 12:02:57 MSK
one should receive
  Last-Modified: Tue, 29 Jun 2021 09:02:57 GMT

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-06-29 12:25:14 +03:00
Denis Kirillov
5eed0fd22d [#67] Added streaming to download
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-24 13:06:13 +03:00
Denis Kirillov
9a49e2c85e [#65] Supported object Content-Type
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-21 16:56:51 +03:00
Denis Kirillov
fffb0b2f12 [#63] Using client.Object from sdk
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-15 11:28:30 +03:00
Roman Khimov
e177e1c9ae *: use client.Object where appropriate
It's enough to do the job, we don't really need full client.Client interface
here.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-06-11 11:54:30 +03:00
Angira Kekteeva
2f8ab7cb8d [#57] downloader, uploader: Replace deprecated
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-06-04 15:55:56 +03:00
Roman Khimov
e8b94553c3 *: update to use unneofsed sdk-go
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-31 23:07:01 +03:00
Angira Kekteeva
82b2126bfd [#46] *: Remove moved to sdk packages, refactoring
Removed connections, logger, neofs because they were moved to sdk repo.
Made changes in downloader, uploader, main.go and app.go via
refactoring of neofs.
Replaced dependencies to removed packages by sdk packages.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-28 14:45:46 +03:00
Angira Kekteeva
33b1a28bf1 Replace http-gate by http-gw in files
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-18 15:14:36 +03:00
Roman Khimov
df3c87af79 *: fix all comment-related golint warnings
Some of this code is going to be moved to SDK library, so it's important.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-13 18:57:42 +03:00
Roman Khimov
293debe6db downloader: ensure downloader/uploader header symmetry
Use X-Attribute-* everywhere.
2021-04-30 14:31:14 +03:00
Roman Khimov
7a46917781 downloader: limit headers sent to the client to some valid subset
fasthttp doesn't do complete filtering/escaping for us, thus filter here.
2021-04-30 14:31:14 +03:00
Roman Khimov
91c894cb8c downloader: set object/container/owner x-attributes after other ones
Avoid name clashes with regular attributes.
2021-04-30 14:31:14 +03:00