efuntikov1337/frostfs-http-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-http-gw
Code Issues Pull requests Projects Releases Packages Wiki Activity
147 commits 2 branches 0 tags 791 KiB
Commit graph

147 commits

This branch
This branch
All branches
Author SHA1 Message Date
Roman Khimov
88e0963240 go.mod: temporarily use nspcc-dev fork of valyala/fasthttp 
We need proper chunk processing.
 2021-04-30 14:31:14 +03:00
Roman Khimov
ed3c576b70 uploader: drain body buffer before ending request processing 
Fixes

2021-04-28T00:03:36.514+0300    debug   uploader/upload.go:64   close temporary multipart/form file     {"cid": "Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ", "address": "Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocX", "filename": "vid.mp4"}
2021/04/28 00:03:36 error when serving connection "127.0.0.1:8082"<->"127.0.0.1:41390": error when reading request headers: cannot find http request method in "0\r\n\r\n". Buffer size=5, contents: "0\r\n\r\n"
 2021-04-30 14:31:14 +03:00
Roman Khimov
b94bfb3b76 app: drop FIXME comment 
We've made long way since a428a0b1b3 and it's
just no longer relevant, the fix is there in fasthttp version we use and
streaming works fine.
 2021-04-30 14:31:14 +03:00
Roman Khimov
464a0134bd neofs: drop PrepareObjectOnsite option 
It's using NeoFS node internal packages which shouldn't ever be used in gate
code, it's not adding any value to the code we have (standard api-go
interfaces work fine).
 2021-04-30 14:31:14 +03:00
Roman Khimov
b020cd6652 uploader: fix passing attributes to object 2021-04-30 14:31:14 +03:00
Roman Khimov
ed27e28a30 app: drop unused Worker/jobDone 2021-04-30 14:31:14 +03:00
Roman Khimov
491ae13190 app: add support for TLS key/cert options 
Run in TLS mode if anything is specified.
 2021-04-15 20:24:49 +03:00
Roman Khimov
a6f63c2bac app: use 1 for weight if not specified or wrong 
If we have a number of nodes with unspecified weights they'd all be treated
equal which seems to be fair.
 2021-04-15 09:45:43 +03:00
Roman Khimov
f019c97ce8 connections: normalize weights, make config a bit more human-oriented 
Current scheme requires people to calculate exact percentages for their
nodes. People are bad with pecentage calculations and it's really inconvenient
if you have more than 3-4 nodes. What I'd like to have is ability to specify
weights like weights, not percentages:
  .._0_WEIGHT=3
  .._1_WEIGHT=1
  .._2_WEIGHT=1

and let the gateway calculate things for me.
 2021-04-15 09:45:43 +03:00
Roman Khimov
0d21ca382f neofs: drop WIF() from Credentials 
It's never used and never needs to be used, the key shouldn't be exported from
the gate.
 2021-04-14 23:52:59 +03:00
Roman Khimov
c06a3137e5 app: add support for ephemeral keys 2021-04-14 23:52:59 +03:00
Roman Khimov
1156223231 go.sum: clean up 
Make this thing build without go.sum changes.
 2021-04-13 12:54:31 +03:00
Stanislav Bogatyrev
b2e7a2cb61 Clean up Makefile and image build 
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
 2021-04-12 17:09:39 +03:00
Pavel Korotkov
e3dbecf844 Rework connection artifacts choice logic 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-08 12:39:29 +03:00
Pavel Korotkov
fad05b76d4 Put artifacts into pool solely 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-08 12:39:29 +03:00
Pavel Korotkov
d7617110b7 Remove test dot-env file 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
c21324bf77 Make minor refactoring 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
f7007f2085 Add connection pool implementation (part 3) 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
a44551d42b Add connection pool implementation (part 2) 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
62a03251ce Add connection pool implementation (part 1) 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
c909c99f72 [#19] Rename client-oriented entities before adding fake/mock ones 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
aedd468c18 [#19] Move connection list sorting to a proper place 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
0b6e138270 [#19] Prepare using connection pool 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
3a5d9fe94c [#19] Extract downloading logic into a separate package 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
eb92219e14 [#19] Extract uploading logic into a separate package 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
4c96885a42 [#19] Add a version with no cdn-sdk deps 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
cdab794d62 Remove sdk context dep 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
f77c4e49dc [#18] Add fetching owner variously on file uploading 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-02-16 17:32:57 +01:00
Pavel Korotkov
237c247ec4 [#13] Add bearer token usage in receive/upload methods 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
e45e5ed6f9 Refactoring and add tests 
- rename `header.go` to `bearer.go`
- simplify header / cookie fetching of bearer token value
- simplify `BearerToken` method
- increase tests

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
b8d10122a9 Add method to fetch BearerToken from Header / Cookies 
- `BearerToken` allow to parse header/cookies to fetch bearer token
- `headerAuth` allow to parse header to fetch bearer token
- `cookieAuth` allow to parse header to fetch bearer token

closes #2
closes #11

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
86991baa33
Merge pull request #14 from nspcc-dev/add-post-method-to-upload-files 
Added POST method to upload files into NeoFS through HTTP Gate
 2021-02-15 15:48:37 +03:00
Evgeniy Kulikov
3f635a018a
Refactoring uploading 
- DisablePreParseMultipartForm = true
- used `fetchMultipartFile` method instead of `MultipartForm`
- don't store temporary files, only streaming

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-13 19:25:12 +03:00
Evgeniy Kulikov
3b8bf3017d
Migrate to fasthttp v1.20.0 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 18:48:52 +03:00
Evgeniy Kulikov
a428a0b1b3
fix conflicts 2021-02-12 15:24:52 +03:00
Evgeniy Kulikov
cbaf9e6142
Fixes after review 
After discussion, we decided to simplify attribute translation for now. Full translation requires support for UTF-8 values encoded according to RFC 8187/7230, this can be clarified and implemented in further PRs.

1. Remove translation tables
2. Use simple translation rules only, for now ignoring UTF-8 keys and values

- `X-Attribute-NEOFS-` prefixed headers considered well-known system attributes
- System Attribute key is uppercased
- System Attribute key's `-` translates to `_`
- `X-Attribute-` prefixed headers considered regular object attributes
- Normal attribute key is a result of http header prefix trimming
- Value string should be left as is, for now

```
HTTP:
X-Attribute-NEOFS-Expiration-Epoch: 123
X-Attribute-FileName: cat.jpg

NeoFS:
__NEOFS__EXPIRATION_EPOCH: "123"
FileName: "cat.jpg"
```

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
71999a796d
Fix logic with set attributes if not set from header 
- if NOT set Filename from header - set from multipart/form
- if NOT set Timestamp from header and enabled by settings - set current timestamp

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
ec70bfa4cc
Fixes after review 
- It's confusing. If there is no difference, I suggest having the route named after the protocol verb or resulting handler. So it should be either post or upload.
- Don't you find that it would be more understandable without else ifs?
- Why do we need a temporary file here?
- etc

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
3cbd4dbd09
Fixes after review 
- Maybe it could be rephrased to "attribute name prefixed with X-Attribute-"
- Maybe it worth renaming to userAttributeHeaderPrefix
- Requirement for having only one file uploaded at a time has to be reflected in documentation.
- Maybe this should be reflected in doc
- I'm not sure if missing timestamp attribute should be forced on gateway level. Normally it should be set by app and there should be a way not to set it.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
ace31ceefd
Add application/json Content-Type header 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
0346db462b
Add cleanup multipart form on defer 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
462de45e12
Added POST method to upload files into NeoFS through HTTP Gate 
- Updated README
- Added method to upload files into NeoFS
- HTTP Upload Header Filter loaded from settings
- Added `HeaderFilter` that filters headers (object attributes)

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
67684c5d8a
Merge pull request #17 from nspcc-dev/update-dependencies 
Update dependencies
 2021-02-12 15:19:13 +03:00
Evgeniy Kulikov
d09a1643e1
Update dependencies 
- update dependencies
  - github.com/nspcc-dev/cdn-sdk v0.3.4
  - github.com/nspcc-dev/neofs-api-go v1.23.0
  - github.com/valyala/fasthttp v1.20.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 12:19:43 +03:00
Evgeniy Kulikov
aa6bb31f4a
Merge pull request #16 from nspcc-dev/update-makefile-and-targets 
Update Makefile and targets
 2021-02-08 11:34:54 +03:00
Evgeniy Kulikov
3b8d684e7c
Update README 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-05 17:13:37 +03:00
Evgeniy Kulikov
dc7aa694b8
Update Makefile and targets 
- add `help.mk` to show information about all targets
- update Makefile

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-05 16:51:20 +03:00
Evgeniy Kulikov
269505c989
Merge pull request #15 from nspcc-dev/migrate-to-sdk-0.3.3 
Update NeoFS API an SDK for HTTP GW
 2021-02-01 19:49:30 +03:00
Evgeniy Kulikov
c2c225988d
Update NeoFS API an SDK for HTTP GW 
- github.com/nspcc-dev/cdn-sdk v0.3.3
- github.com/nspcc-dev/neofs-api-go v1.22.2

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-01 17:31:50 +03:00
Evgeniy Kulikov
b6d94ec9a5
Fixes around get_by_attribute 
Before, `attr_val` ignores attribute values that contains slashes in it value, now it's fixed.

Update to v0.3.2 with fixing status of connection pool.

Added debug log message when object not found, for debug reasons.

Update .test.env variables prefix, it was changed in one of previous releases.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-01-23 16:28:12 +03:00