Commit graph

60 commits

Author SHA1 Message Date
Angira Kekteeva
d8b04223d4 [#186] Improve wallet params
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-03 17:57:48 +03:00
Denis Kirillov
ee49355bb7 [#179] Expose pool metrics
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-02 17:26:11 +03:00
Denis Kirillov
0f7737088d [#184] Add config param for pool error threshold
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-29 11:23:44 +03:00
Denis Kirillov
af732d294c [#171] Sync metrics and pprof configuration
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-29 09:49:57 +03:00
Denis Kirillov
0fb75a1cde [#178] Update SDK to v1.0.0-rc.5
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-25 14:27:24 +03:00
Denis Kirillov
cf018c2fab [#175] Add health metric
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-22 17:17:36 +03:00
Denis Kirillov
a42606742a [#142] Support resolving container nicename
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-28 17:18:13 +03:00
Alex Vanin
1e3df95eed [#145] Use application context in NeoFS API requests
It is meaningless to use RequestCtx as a context.Context
for NeoFS operation, because context won't be closed
until application shutdown. Moreover, it also triggers
data race detection, because server's done channel, which
is accessible for reading from RequestCtx, is set to `nil`.

Using application context doesn't change gateway behavior,
but it suppresses data race trigger at shutdown. It also
allows possibility to set configurable timeouts for NeoFS
networking if we will ever need them.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-22 15:33:11 +04:00
Denis Kirillov
eacd7be33e [#140] Improve docs
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-15 03:26:02 +04:00
Denis Kirillov
baf425453d [#139] fix lint
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-08 00:23:12 +04:00
Denis Kirillov
5080b43a04 [#139] Update SDK
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-04-08 00:23:12 +04:00
Denis Kirillov
d891c13cb3 [#137] Drop sdk logger
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-29 10:15:59 +03:00
Denis Kirillov
6d362894ad [#132] Update NeoFS SDK to v1.0.0-rc.3
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-03-16 17:15:21 +03:00
Leonard Lyubich
2b7e4a36fb [#126] Upgrade NeoFS SDK Go library
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-03 18:25:53 +03:00
Denis Kirillov
71e846a4fc [#115] Update sdk version to support priority
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-12-10 10:59:43 +03:00
Angira Kekteeva
06be798111 [#109] Add version to logs in start
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-11-29 13:56:36 +03:00
Alex Vanin
79765fb7cc [#106] Update neofs-sdk-go to the latest version
Includes:
 - fix for unavailable endpoints of client pool,
 - session token cache with request retry in case
   of missing session token,
 - migrated neofs client.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-15 14:46:50 +03:00
Denis Kirillov
6265fcb26a [#105] Add newline to errors
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-12 15:32:18 +03:00
Denis Kirillov
9b92c0146c [#96] Add zip compression
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-11-01 11:27:22 +03:00
Denis Kirillov
0b364504a7 [#92] Support zip download
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-10-22 18:36:09 +03:00
Denis Kirillov
d856fdf4c4 [#76] Added HEAD methods
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-14 18:56:27 +03:00
Denis Kirillov
2b8aa4914e [#77] Added requests logging
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-07-09 13:48:50 +03:00
Denis Kirillov
2d41fbc95e [#68] Support NEP-6 wallets
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-23 14:53:04 +03:00
Denis Kirillov
7734be609c [#60] Dropped grpc keepalive options
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2021-06-10 18:48:55 +03:00
Roman Khimov
e8b94553c3 *: update to use unneofsed sdk-go
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-31 23:07:01 +03:00
Angira Kekteeva
82b2126bfd [#46] *: Remove moved to sdk packages, refactoring
Removed connections, logger, neofs because they were moved to sdk repo.
Made changes in downloader, uploader, main.go and app.go via
refactoring of neofs.
Replaced dependencies to removed packages by sdk packages.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-28 14:45:46 +03:00
Angira Kekteeva
33b1a28bf1 Replace http-gate by http-gw in files
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-18 15:14:36 +03:00
Roman Khimov
df3c87af79 *: fix all comment-related golint warnings
Some of this code is going to be moved to SDK library, so it's important.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-13 18:57:42 +03:00
Roman Khimov
f99f9e88a7 connections: make use of keepalive options 2021-04-30 14:31:14 +03:00
Roman Khimov
b94bfb3b76 app: drop FIXME comment
We've made long way since a428a0b1b3 and it's
just no longer relevant, the fix is there in fasthttp version we use and
streaming works fine.
2021-04-30 14:31:14 +03:00
Roman Khimov
ed27e28a30 app: drop unused Worker/jobDone 2021-04-30 14:31:14 +03:00
Roman Khimov
491ae13190 app: add support for TLS key/cert options
Run in TLS mode if anything is specified.
2021-04-15 20:24:49 +03:00
Roman Khimov
a6f63c2bac app: use 1 for weight if not specified or wrong
If we have a number of nodes with unspecified weights they'd all be treated
equal which seems to be fair.
2021-04-15 09:45:43 +03:00
Roman Khimov
c06a3137e5 app: add support for ephemeral keys 2021-04-14 23:52:59 +03:00
Pavel Korotkov
fad05b76d4 Put artifacts into pool solely
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-08 12:39:29 +03:00
Pavel Korotkov
62a03251ce Add connection pool implementation (part 1)
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
aedd468c18 [#19] Move connection list sorting to a proper place
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
0b6e138270 [#19] Prepare using connection pool
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
3a5d9fe94c [#19] Extract downloading logic into a separate package
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
eb92219e14 [#19] Extract uploading logic into a separate package
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
4c96885a42 [#19] Add a version with no cdn-sdk deps
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Evgeniy Kulikov
3f635a018a
Refactoring uploading
- DisablePreParseMultipartForm = true
- used `fetchMultipartFile` method instead of `MultipartForm`
- don't store temporary files, only streaming

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-13 19:25:12 +03:00
Evgeniy Kulikov
3b8bf3017d
Migrate to fasthttp v1.20.0
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 18:48:52 +03:00
Evgeniy Kulikov
a428a0b1b3
fix conflicts 2021-02-12 15:24:52 +03:00
Evgeniy Kulikov
cbaf9e6142
Fixes after review
After discussion, we decided to simplify attribute translation for now. Full translation requires support for UTF-8 values encoded according to RFC 8187/7230, this can be clarified and implemented in further PRs.

1. Remove translation tables
2. Use simple translation rules only, for now ignoring UTF-8 keys and values

- `X-Attribute-NEOFS-` prefixed headers considered well-known system attributes
- System Attribute key is uppercased
- System Attribute key's `-` translates to `_`
- `X-Attribute-` prefixed headers considered regular object attributes
- Normal attribute key is a result of http header prefix trimming
- Value string should be left as is, for now

```
HTTP:
X-Attribute-NEOFS-Expiration-Epoch: 123
X-Attribute-FileName: cat.jpg

NeoFS:
__NEOFS__EXPIRATION_EPOCH: "123"
FileName: "cat.jpg"
```

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
ec70bfa4cc
Fixes after review
- It's confusing. If there is no difference, I suggest having the route named after the protocol verb or resulting handler. So it should be either post or upload.
- Don't you find that it would be more understandable without else ifs?
- Why do we need a temporary file here?
- etc

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
3cbd4dbd09
Fixes after review
- Maybe it could be rephrased to "attribute name prefixed with X-Attribute-"
- Maybe it worth renaming to userAttributeHeaderPrefix
- Requirement for having only one file uploaded at a time has to be reflected in documentation.
- Maybe this should be reflected in doc
- I'm not sure if missing timestamp attribute should be forced on gateway level. Normally it should be set by app and there should be a way not to set it.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
462de45e12
Added POST method to upload files into NeoFS through HTTP Gate
- Updated README
- Added method to upload files into NeoFS
- HTTP Upload Header Filter loaded from settings
- Added `HeaderFilter` that filters headers (object attributes)

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
b6d94ec9a5
Fixes around get_by_attribute
Before, `attr_val` ignores attribute values that contains slashes in it value, now it's fixed.

Update to v0.3.2 with fixing status of connection pool.

Added debug log message when object not found, for debug reasons.

Update .test.env variables prefix, it was changed in one of previous releases.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-01-23 16:28:12 +03:00
Evgeniy Kulikov
c30b264beb
Migrate to SDK 0.3.0
- update dependencies
  - github.com/fasthttp/router v1.3.5
  - github.com/nspcc-dev/cdn-sdk v0.3.0
  - github.com/nspcc-dev/neofs-api-go v1.22.0
  - github.com/prometheus/client_golang v1.9.0
  - github.com/valyala/fasthttp v1.19.0
  - google.golang.org/grpc v1.35.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-01-14 13:31:40 +03:00