pogpp/frostfs-http-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-http-gw
Code Pull requests Activity
248 commits 4 branches 1 tag 1.6 MiB
Commit graph

248 commits

This branch
This branch
All branches
Author SHA1 Message Date
Roman Khimov
4e4f873786 misc: gofmt -s 2021-04-30 14:31:14 +03:00
Roman Khimov
88e0963240 go.mod: temporarily use nspcc-dev fork of valyala/fasthttp 
We need proper chunk processing.
 2021-04-30 14:31:14 +03:00
Roman Khimov
ed3c576b70 uploader: drain body buffer before ending request processing 
Fixes

2021-04-28T00:03:36.514+0300    debug   uploader/upload.go:64   close temporary multipart/form file     {"cid": "Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ", "address": "Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocX", "filename": "vid.mp4"}
2021/04/28 00:03:36 error when serving connection "127.0.0.1:8082"<->"127.0.0.1:41390": error when reading request headers: cannot find http request method in "0\r\n\r\n". Buffer size=5, contents: "0\r\n\r\n"
 2021-04-30 14:31:14 +03:00
Roman Khimov
b94bfb3b76 app: drop FIXME comment 
We've made long way since a428a0b1b3 and it's
just no longer relevant, the fix is there in fasthttp version we use and
streaming works fine.
 2021-04-30 14:31:14 +03:00
Roman Khimov
464a0134bd neofs: drop PrepareObjectOnsite option 
It's using NeoFS node internal packages which shouldn't ever be used in gate
code, it's not adding any value to the code we have (standard api-go
interfaces work fine).
 2021-04-30 14:31:14 +03:00
Roman Khimov
b020cd6652 uploader: fix passing attributes to object 2021-04-30 14:31:14 +03:00
Roman Khimov
ed27e28a30 app: drop unused Worker/jobDone 2021-04-30 14:31:14 +03:00
Roman Khimov
491ae13190 app: add support for TLS key/cert options 
Run in TLS mode if anything is specified.
 2021-04-15 20:24:49 +03:00
Roman Khimov
a6f63c2bac app: use 1 for weight if not specified or wrong 
If we have a number of nodes with unspecified weights they'd all be treated
equal which seems to be fair.
 2021-04-15 09:45:43 +03:00
Roman Khimov
f019c97ce8 connections: normalize weights, make config a bit more human-oriented 
Current scheme requires people to calculate exact percentages for their
nodes. People are bad with pecentage calculations and it's really inconvenient
if you have more than 3-4 nodes. What I'd like to have is ability to specify
weights like weights, not percentages:
  .._0_WEIGHT=3
  .._1_WEIGHT=1
  .._2_WEIGHT=1

and let the gateway calculate things for me.
 2021-04-15 09:45:43 +03:00
Roman Khimov
0d21ca382f neofs: drop WIF() from Credentials 
It's never used and never needs to be used, the key shouldn't be exported from
the gate.
 2021-04-14 23:52:59 +03:00
Roman Khimov
c06a3137e5 app: add support for ephemeral keys 2021-04-14 23:52:59 +03:00
Roman Khimov
1156223231 go.sum: clean up 
Make this thing build without go.sum changes.
 2021-04-13 12:54:31 +03:00
Stanislav Bogatyrev
b2e7a2cb61 Clean up Makefile and image build 
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
 2021-04-12 17:09:39 +03:00
Pavel Korotkov
e3dbecf844 Rework connection artifacts choice logic 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-08 12:39:29 +03:00
Pavel Korotkov
fad05b76d4 Put artifacts into pool solely 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-08 12:39:29 +03:00
Pavel Korotkov
d7617110b7 Remove test dot-env file 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
c21324bf77 Make minor refactoring 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
f7007f2085 Add connection pool implementation (part 3) 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
a44551d42b Add connection pool implementation (part 2) 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
62a03251ce Add connection pool implementation (part 1) 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
c909c99f72 [#19] Rename client-oriented entities before adding fake/mock ones 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
aedd468c18 [#19] Move connection list sorting to a proper place 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
0b6e138270 [#19] Prepare using connection pool 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
3a5d9fe94c [#19] Extract downloading logic into a separate package 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
eb92219e14 [#19] Extract uploading logic into a separate package 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
4c96885a42 [#19] Add a version with no cdn-sdk deps 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
cdab794d62 Remove sdk context dep 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
f77c4e49dc [#18] Add fetching owner variously on file uploading 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-02-16 17:32:57 +01:00
Pavel Korotkov
237c247ec4 [#13] Add bearer token usage in receive/upload methods 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
e45e5ed6f9 Refactoring and add tests 
- rename `header.go` to `bearer.go`
- simplify header / cookie fetching of bearer token value
- simplify `BearerToken` method
- increase tests

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
b8d10122a9 Add method to fetch BearerToken from Header / Cookies 
- `BearerToken` allow to parse header/cookies to fetch bearer token
- `headerAuth` allow to parse header to fetch bearer token
- `cookieAuth` allow to parse header to fetch bearer token

closes #2
closes #11

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
86991baa33
Merge pull request #14 from nspcc-dev/add-post-method-to-upload-files 
Added POST method to upload files into NeoFS through HTTP Gate
 2021-02-15 15:48:37 +03:00
Evgeniy Kulikov
3f635a018a
Refactoring uploading 
- DisablePreParseMultipartForm = true
- used `fetchMultipartFile` method instead of `MultipartForm`
- don't store temporary files, only streaming

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-13 19:25:12 +03:00
Evgeniy Kulikov
3b8bf3017d
Migrate to fasthttp v1.20.0 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 18:48:52 +03:00
Evgeniy Kulikov
a428a0b1b3
fix conflicts 2021-02-12 15:24:52 +03:00
Evgeniy Kulikov
cbaf9e6142
Fixes after review 
After discussion, we decided to simplify attribute translation for now. Full translation requires support for UTF-8 values encoded according to RFC 8187/7230, this can be clarified and implemented in further PRs.

1. Remove translation tables
2. Use simple translation rules only, for now ignoring UTF-8 keys and values

- `X-Attribute-NEOFS-` prefixed headers considered well-known system attributes
- System Attribute key is uppercased
- System Attribute key's `-` translates to `_`
- `X-Attribute-` prefixed headers considered regular object attributes
- Normal attribute key is a result of http header prefix trimming
- Value string should be left as is, for now

```
HTTP:
X-Attribute-NEOFS-Expiration-Epoch: 123
X-Attribute-FileName: cat.jpg

NeoFS:
__NEOFS__EXPIRATION_EPOCH: "123"
FileName: "cat.jpg"
```

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
71999a796d
Fix logic with set attributes if not set from header 
- if NOT set Filename from header - set from multipart/form
- if NOT set Timestamp from header and enabled by settings - set current timestamp

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
ec70bfa4cc
Fixes after review 
- It's confusing. If there is no difference, I suggest having the route named after the protocol verb or resulting handler. So it should be either post or upload.
- Don't you find that it would be more understandable without else ifs?
- Why do we need a temporary file here?
- etc

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
3cbd4dbd09
Fixes after review 
- Maybe it could be rephrased to "attribute name prefixed with X-Attribute-"
- Maybe it worth renaming to userAttributeHeaderPrefix
- Requirement for having only one file uploaded at a time has to be reflected in documentation.
- Maybe this should be reflected in doc
- I'm not sure if missing timestamp attribute should be forced on gateway level. Normally it should be set by app and there should be a way not to set it.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
ace31ceefd
Add application/json Content-Type header 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
0346db462b
Add cleanup multipart form on defer 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
462de45e12
Added POST method to upload files into NeoFS through HTTP Gate 
- Updated README
- Added method to upload files into NeoFS
- HTTP Upload Header Filter loaded from settings
- Added `HeaderFilter` that filters headers (object attributes)

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
67684c5d8a
Merge pull request #17 from nspcc-dev/update-dependencies 
Update dependencies
 2021-02-12 15:19:13 +03:00
Evgeniy Kulikov
d09a1643e1
Update dependencies 
- update dependencies
  - github.com/nspcc-dev/cdn-sdk v0.3.4
  - github.com/nspcc-dev/neofs-api-go v1.23.0
  - github.com/valyala/fasthttp v1.20.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 12:19:43 +03:00
Evgeniy Kulikov
aa6bb31f4a
Merge pull request #16 from nspcc-dev/update-makefile-and-targets 
Update Makefile and targets
 2021-02-08 11:34:54 +03:00
Evgeniy Kulikov
3b8d684e7c
Update README 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-05 17:13:37 +03:00
Evgeniy Kulikov
dc7aa694b8
Update Makefile and targets 
- add `help.mk` to show information about all targets
- update Makefile

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-05 16:51:20 +03:00
Evgeniy Kulikov
269505c989
Merge pull request #15 from nspcc-dev/migrate-to-sdk-0.3.3 
Update NeoFS API an SDK for HTTP GW
 2021-02-01 19:49:30 +03:00
Evgeniy Kulikov
c2c225988d
Update NeoFS API an SDK for HTTP GW 
- github.com/nspcc-dev/cdn-sdk v0.3.3
- github.com/nspcc-dev/neofs-api-go v1.22.2

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-01 17:31:50 +03:00