dkirillov/frostfs-http-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-http-gw
Code Issues Pull requests Projects Releases Packages Wiki Activity
229 commits 4 branches 0 tags 2.6 MiB
Commit graph

229 commits

This branch
This branch
All branches
Author SHA1 Message Date
Pavel Korotkov
a44551d42b Add connection pool implementation (part 2) 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
62a03251ce Add connection pool implementation (part 1) 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
c909c99f72 [#19] Rename client-oriented entities before adding fake/mock ones 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
aedd468c18 [#19] Move connection list sorting to a proper place 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
0b6e138270 [#19] Prepare using connection pool 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
3a5d9fe94c [#19] Extract downloading logic into a separate package 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
eb92219e14 [#19] Extract uploading logic into a separate package 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
4c96885a42 [#19] Add a version with no cdn-sdk deps 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
cdab794d62 Remove sdk context dep 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-04-06 12:19:21 +03:00
Pavel Korotkov
f77c4e49dc [#18] Add fetching owner variously on file uploading 
Signed-off-by: Pavel Korotkov <pavel@nspcc.ru>
 2021-02-16 17:32:57 +01:00
Pavel Korotkov
237c247ec4 [#13] Add bearer token usage in receive/upload methods 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
e45e5ed6f9 Refactoring and add tests 
- rename `header.go` to `bearer.go`
- simplify header / cookie fetching of bearer token value
- simplify `BearerToken` method
- increase tests

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
b8d10122a9 Add method to fetch BearerToken from Header / Cookies 
- `BearerToken` allow to parse header/cookies to fetch bearer token
- `headerAuth` allow to parse header to fetch bearer token
- `cookieAuth` allow to parse header to fetch bearer token

closes #2
closes #11

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-16 16:22:25 +01:00
Evgeniy Kulikov
86991baa33
Merge pull request #14 from nspcc-dev/add-post-method-to-upload-files 
Added POST method to upload files into NeoFS through HTTP Gate
 2021-02-15 15:48:37 +03:00
Evgeniy Kulikov
3f635a018a
Refactoring uploading 
- DisablePreParseMultipartForm = true
- used `fetchMultipartFile` method instead of `MultipartForm`
- don't store temporary files, only streaming

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-13 19:25:12 +03:00
Evgeniy Kulikov
3b8bf3017d
Migrate to fasthttp v1.20.0 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 18:48:52 +03:00
Evgeniy Kulikov
a428a0b1b3
fix conflicts 2021-02-12 15:24:52 +03:00
Evgeniy Kulikov
cbaf9e6142
Fixes after review 
After discussion, we decided to simplify attribute translation for now. Full translation requires support for UTF-8 values encoded according to RFC 8187/7230, this can be clarified and implemented in further PRs.

1. Remove translation tables
2. Use simple translation rules only, for now ignoring UTF-8 keys and values

- `X-Attribute-NEOFS-` prefixed headers considered well-known system attributes
- System Attribute key is uppercased
- System Attribute key's `-` translates to `_`
- `X-Attribute-` prefixed headers considered regular object attributes
- Normal attribute key is a result of http header prefix trimming
- Value string should be left as is, for now

```
HTTP:
X-Attribute-NEOFS-Expiration-Epoch: 123
X-Attribute-FileName: cat.jpg

NeoFS:
__NEOFS__EXPIRATION_EPOCH: "123"
FileName: "cat.jpg"
```

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
71999a796d
Fix logic with set attributes if not set from header 
- if NOT set Filename from header - set from multipart/form
- if NOT set Timestamp from header and enabled by settings - set current timestamp

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
ec70bfa4cc
Fixes after review 
- It's confusing. If there is no difference, I suggest having the route named after the protocol verb or resulting handler. So it should be either post or upload.
- Don't you find that it would be more understandable without else ifs?
- Why do we need a temporary file here?
- etc

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
3cbd4dbd09
Fixes after review 
- Maybe it could be rephrased to "attribute name prefixed with X-Attribute-"
- Maybe it worth renaming to userAttributeHeaderPrefix
- Requirement for having only one file uploaded at a time has to be reflected in documentation.
- Maybe this should be reflected in doc
- I'm not sure if missing timestamp attribute should be forced on gateway level. Normally it should be set by app and there should be a way not to set it.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
ace31ceefd
Add application/json Content-Type header 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
0346db462b
Add cleanup multipart form on defer 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
462de45e12
Added POST method to upload files into NeoFS through HTTP Gate 
- Updated README
- Added method to upload files into NeoFS
- HTTP Upload Header Filter loaded from settings
- Added `HeaderFilter` that filters headers (object attributes)

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 15:21:53 +03:00
Evgeniy Kulikov
67684c5d8a
Merge pull request #17 from nspcc-dev/update-dependencies 
Update dependencies
 2021-02-12 15:19:13 +03:00
Evgeniy Kulikov
d09a1643e1
Update dependencies 
- update dependencies
  - github.com/nspcc-dev/cdn-sdk v0.3.4
  - github.com/nspcc-dev/neofs-api-go v1.23.0
  - github.com/valyala/fasthttp v1.20.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-12 12:19:43 +03:00
Evgeniy Kulikov
aa6bb31f4a
Merge pull request #16 from nspcc-dev/update-makefile-and-targets 
Update Makefile and targets
 2021-02-08 11:34:54 +03:00
Evgeniy Kulikov
3b8d684e7c
Update README 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-05 17:13:37 +03:00
Evgeniy Kulikov
dc7aa694b8
Update Makefile and targets 
- add `help.mk` to show information about all targets
- update Makefile

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-05 16:51:20 +03:00
Evgeniy Kulikov
269505c989
Merge pull request #15 from nspcc-dev/migrate-to-sdk-0.3.3 
Update NeoFS API an SDK for HTTP GW
 2021-02-01 19:49:30 +03:00
Evgeniy Kulikov
c2c225988d
Update NeoFS API an SDK for HTTP GW 
- github.com/nspcc-dev/cdn-sdk v0.3.3
- github.com/nspcc-dev/neofs-api-go v1.22.2

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-02-01 17:31:50 +03:00
Evgeniy Kulikov
b6d94ec9a5
Fixes around get_by_attribute 
Before, `attr_val` ignores attribute values that contains slashes in it value, now it's fixed.

Update to v0.3.2 with fixing status of connection pool.

Added debug log message when object not found, for debug reasons.

Update .test.env variables prefix, it was changed in one of previous releases.

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-01-23 16:28:12 +03:00
Evgeniy Kulikov
52812f7869
Release v0.14.1 with CDN SDK v0.3.1 and NeoFS API Go v1.22.1 2021-01-18 10:08:44 +03:00
Evgeniy Kulikov
ccf82b34db
Merge pull request #9 from nspcc-dev/update-to-new-sdk-release 
Migrate to SDK 0.3.0
 2021-01-14 13:35:17 +03:00
Evgeniy Kulikov
c30b264beb
Migrate to SDK 0.3.0 
- update dependencies
  - github.com/fasthttp/router v1.3.5
  - github.com/nspcc-dev/cdn-sdk v0.3.0
  - github.com/nspcc-dev/neofs-api-go v1.22.0
  - github.com/prometheus/client_golang v1.9.0
  - github.com/valyala/fasthttp v1.19.0
  - google.golang.org/grpc v1.35.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-01-14 13:31:40 +03:00
Evgeniy Kulikov
ac8affff91
Migrate to new release SDK 2020-12-23 14:21:34 +03:00
Evgeniy Kulikov
cb1b76a103
Merge pull request #7 from nspcc-dev/how-to-check-http-gate 
Add how to check HTTP Gate
 2020-12-04 09:56:44 +03:00
Evgeniy Kulikov
1b0acd4836
Merge pull request #6 from nspcc-dev/refactoring_http_gate 
Refactoring HTTP Gate
 2020-12-04 09:56:12 +03:00
Evgeniy Kulikov
4fb7e511dc
Rename how-to-check.mc to how-to-check.md 2020-12-03 19:12:00 +03:00
Evgeniy Kulikov
abd95efe47
Refactoring HTTP Gate 
- simplify code
- add `/get_by_attribute`
- refactoring receiving object

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-03 18:50:35 +03:00
Evgeniy Kulikov
77c404d78f
Add how to check HTTP Gate 2020-12-03 18:46:36 +03:00
Evgeniy Kulikov
b6a9654546
Merge pull request #5 from nspcc-dev/prepare-to-release 
Prepare to release
 2020-12-02 13:34:32 +03:00
Evgeniy Kulikov
a18e4200b0
Cleanup dependencies 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-02 12:25:18 +03:00
Evgeniy Kulikov
65a5a3f9fc
Update README 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-02 12:25:18 +03:00
Evgeniy Kulikov
8bd210d67e
Use constant settings keys 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-02 12:25:18 +03:00
Evgeniy Kulikov
01f13ef210
Prepare to release 
- set output for flag set
- change env prefix for all environments
- migrate to CDN SDK release v0.1.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-02 12:25:18 +03:00
Evgeniy Kulikov
420531d12a
Merge pull request #4 from nspcc-dev/migrate-to-cdn-sdk 
Migrate to new release of NeoFS API
 2020-12-02 12:25:00 +03:00
Evgeniy Kulikov
31186382e8
Change ENV prefix 
TODO should be replaced with HTTP_GW before release

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-27 18:18:53 +03:00
Evgeniy Kulikov
f75eb4b803
Update dependencies 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-27 16:38:58 +03:00
Evgeniy Kulikov
3166adc410
Deprecation warning 2020-11-23 12:43:27 +03:00