Commit graph

85 commits

Author SHA1 Message Date
Angira Kekteeva
82b2126bfd [#46] *: Remove moved to sdk packages, refactoring
Removed connections, logger, neofs because they were moved to sdk repo.
Made changes in downloader, uploader, main.go and app.go via
refactoring of neofs.
Replaced dependencies to removed packages by sdk packages.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-28 14:45:46 +03:00
Angira Kekteeva
33b1a28bf1 Replace http-gate by http-gw in files
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-18 15:14:36 +03:00
Roman Khimov
df3c87af79 *: fix all comment-related golint warnings
Some of this code is going to be moved to SDK library, so it's important.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2021-05-13 18:57:42 +03:00
Roman Khimov
f99f9e88a7 connections: make use of keepalive options 2021-04-30 14:31:14 +03:00
Roman Khimov
b94bfb3b76 app: drop FIXME comment
We've made long way since a428a0b1b3 and it's
just no longer relevant, the fix is there in fasthttp version we use and
streaming works fine.
2021-04-30 14:31:14 +03:00
Roman Khimov
ed27e28a30 app: drop unused Worker/jobDone 2021-04-30 14:31:14 +03:00
Roman Khimov
491ae13190 app: add support for TLS key/cert options
Run in TLS mode if anything is specified.
2021-04-15 20:24:49 +03:00
Roman Khimov
a6f63c2bac app: use 1 for weight if not specified or wrong
If we have a number of nodes with unspecified weights they'd all be treated
equal which seems to be fair.
2021-04-15 09:45:43 +03:00
Roman Khimov
c06a3137e5 app: add support for ephemeral keys 2021-04-14 23:52:59 +03:00
Pavel Korotkov
fad05b76d4 Put artifacts into pool solely
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-08 12:39:29 +03:00
Pavel Korotkov
62a03251ce Add connection pool implementation (part 1)
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
aedd468c18 [#19] Move connection list sorting to a proper place
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
0b6e138270 [#19] Prepare using connection pool
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
3a5d9fe94c [#19] Extract downloading logic into a separate package
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
eb92219e14 [#19] Extract uploading logic into a separate package
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Pavel Korotkov
4c96885a42 [#19] Add a version with no cdn-sdk deps
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
2021-04-06 12:19:21 +03:00
Evgeniy Kulikov
3f635a018a
Refactoring uploading
- DisablePreParseMultipartForm = true
- used `fetchMultipartFile` method instead of `MultipartForm`
- don't store temporary files, only streaming

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-13 19:25:12 +03:00
Evgeniy Kulikov
3b8bf3017d
Migrate to fasthttp v1.20.0
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 18:48:52 +03:00
Evgeniy Kulikov
a428a0b1b3
fix conflicts 2021-02-12 15:24:52 +03:00
Evgeniy Kulikov
cbaf9e6142
Fixes after review
After discussion, we decided to simplify attribute translation for now. Full translation requires support for UTF-8 values encoded according to RFC 8187/7230, this can be clarified and implemented in further PRs.

1. Remove translation tables
2. Use simple translation rules only, for now ignoring UTF-8 keys and values

- `X-Attribute-NEOFS-` prefixed headers considered well-known system attributes
- System Attribute key is uppercased
- System Attribute key's `-` translates to `_`
- `X-Attribute-` prefixed headers considered regular object attributes
- Normal attribute key is a result of http header prefix trimming
- Value string should be left as is, for now

```
HTTP:
X-Attribute-NEOFS-Expiration-Epoch: 123
X-Attribute-FileName: cat.jpg

NeoFS:
__NEOFS__EXPIRATION_EPOCH: "123"
FileName: "cat.jpg"
```

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
ec70bfa4cc
Fixes after review
- It's confusing. If there is no difference, I suggest having the route named after the protocol verb or resulting handler. So it should be either post or upload.
- Don't you find that it would be more understandable without else ifs?
- Why do we need a temporary file here?
- etc

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
3cbd4dbd09
Fixes after review
- Maybe it could be rephrased to "attribute name prefixed with X-Attribute-"
- Maybe it worth renaming to userAttributeHeaderPrefix
- Requirement for having only one file uploaded at a time has to be reflected in documentation.
- Maybe this should be reflected in doc
- I'm not sure if missing timestamp attribute should be forced on gateway level. Normally it should be set by app and there should be a way not to set it.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
462de45e12
Added POST method to upload files into NeoFS through HTTP Gate
- Updated README
- Added method to upload files into NeoFS
- HTTP Upload Header Filter loaded from settings
- Added `HeaderFilter` that filters headers (object attributes)

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
b6d94ec9a5
Fixes around get_by_attribute
Before, `attr_val` ignores attribute values that contains slashes in it value, now it's fixed.

Update to v0.3.2 with fixing status of connection pool.

Added debug log message when object not found, for debug reasons.

Update .test.env variables prefix, it was changed in one of previous releases.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-01-23 16:28:12 +03:00
Evgeniy Kulikov
c30b264beb
Migrate to SDK 0.3.0
- update dependencies
  - github.com/fasthttp/router v1.3.5
  - github.com/nspcc-dev/cdn-sdk v0.3.0
  - github.com/nspcc-dev/neofs-api-go v1.22.0
  - github.com/prometheus/client_golang v1.9.0
  - github.com/valyala/fasthttp v1.19.0
  - google.golang.org/grpc v1.35.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2021-01-14 13:31:40 +03:00
Evgeniy Kulikov
abd95efe47
Refactoring HTTP Gate
- simplify code
- add `/get_by_attribute`
- refactoring receiving object

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-12-03 18:50:35 +03:00
Evgeniy Kulikov
8bd210d67e
Use constant settings keys
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-12-02 12:25:18 +03:00
Evgeniy Kulikov
a4f7a2f962
Migrate to new release of NeoFS API
- migrate to new release NeoFS API / CDN SDK
- add `Last-Modified` header

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-11-23 12:32:03 +03:00
Evgeniy Kulikov
6226729e38
Migrate to CDN SDK 2020-11-10 14:46:28 +03:00
Evgeniy Kulikov
bbb2a057c7
NFSSVC-16 Sign health-check requests 2020-07-02 11:34:54 +03:00
Evgeniy Kulikov
799d391508
migrate to new api v1.0.0 2020-06-03 06:30:58 +03:00
Evgeniy Kulikov
21fc1c6e6c
log: fix routes in log message 2020-05-12 11:19:07 +03:00
Evgeniy Kulikov
54071b5da4
routing: remove trailing slash 2020-05-12 11:18:23 +03:00
Evgeniy Kulikov
ea4710eaba
update to new router 2020-04-15 11:10:42 +03:00
Evgeniy Kulikov
164b0870de
refactoring + moved to neofs-api-go 2020-03-31 11:37:10 +03:00