mbiryukova/frostfs-http-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-http-gw
Code Pull requests Activity

[#153] English Check

Browse source 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
This commit is contained in:
Elizaveta Chichindaeva 2022-04-21 11:35:57 +03:00 committed by Alex Vanin
parent 1e3df95eed
commit 11283c1c79
6 changed files with 57 additions and 57 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
CONTRIBUTING.md
View file

@ -9,14 +9,14 @@ everyone. Please follow the guidelines:
- Open an issue first, to discuss a new feature or enhancement.
- Write tests, and make sure the test suite passes locally and on CI.
- Write tests and make sure the test suite passes locally and on CI.
- Open a pull request, and reference the relevant issue(s).
- Open a pull request and reference the relevant issue(s).
- Make sure your commits are logically separated and have good comments
explaining the details of your change.
- After receiving feedback, amend your commits or add new ones as
- After receiving a feedback, amend your commits or add new ones as
appropriate.
- **Have fun!**
@ -48,7 +48,7 @@ $ git merge upstream/master
### Create your feature branch
Before making code changes, make sure you create a separate branch for these
changes. Maybe you will find it convenient to name branch in
changes. Maybe you will find it convenient to name a branch in
`<type>/<Issue>-<changes_topic>` format.
```
@ -98,7 +98,7 @@ reviewed and approved, it will be merged.
## DCO Sign off
All authors to the project retain copyright to their work. However, to ensure
that they are only submitting work that they have rights to, we are requiring
that they are only submitting work that they have rights to, we require
everyone to acknowledge this by signing their work.
Any copyright notices in this repository should specify the authors as "the
@ -110,7 +110,7 @@ To sign your work, just add a line like this at the end of your commit message:
Signed-off-by: Samii Sakisaka <samii@nspcc.ru>
```
This can easily be done with the `--signoff` option to `git commit`.
This can be easily done with the `--signoff` option to `git commit`.
By doing this you state that you can certify the following (from [The Developer
Certificate of Origin](https://developercertificate.org/)):

74
README.md
View file

@ -13,7 +13,7 @@
# NeoFS HTTP Gateway
NeoFS HTTP Gateway bridges NeoFS internal protocol and HTTP standard.
- you can download one file per request from NeoFS Network
- you can download one file per request from the NeoFS Network
- you can upload one file per request into the NeoFS Network
## Installation
@ -35,8 +35,8 @@ version Show current version
```
Or you can also use a [Docker
image](https://hub.docker.com/r/nspccdev/neofs-http-gw) provided for released
(and occasionally unreleased) versions of gateway (`:latest` points to the
image](https://hub.docker.com/r/nspccdev/neofs-http-gw) provided for the released
(and occasionally unreleased) versions of the gateway (`:latest` points to the
latest stable release).
## Execution
@ -47,8 +47,8 @@ can be done either via `-p` parameter or via `HTTP_GW_PEERS_<N>_ADDRESS` and
`HTTP_GW_PEERS_<N>_WEIGHT` environment variables (the gate supports multiple
NeoFS nodes with weighted load balancing).
If you're launching HTTP gateway in bundle with [neofs-dev-env](https://github.com/nspcc-dev/neofs-dev-env),
you can get an IP address of the node in output of `make hosts` command
If you launch HTTP gateway in bundle with [neofs-dev-env](https://github.com/nspcc-dev/neofs-dev-env),
you can get the IP address of the node in the output of `make hosts` command
(with s0*.neofs.devenv name).
These two commands are functionally equivalent, they run the gate with one
@ -89,9 +89,9 @@ This command will make gateway use 192.168.130.71 while it is healthy. Otherwise
192.168.130.72 for 90% of requests and 192.168.130.73 for remaining 10%.
### Keys
You can provide wallet via `--wallet` or `-w` flag also you can specify account address using `--address`
(if no address provided default one will be used). If wallet is used you need to set `HTTP_GW_WALLET_PASSPHRASE` variable to decrypt wallet.
If no wallet provided gateway autogenerates key pair it will use for NeoFS requests.
You can provide a wallet via `--wallet` or `-w` flag. You can also specify the account address using `--address`
(if no address provided default one will be used). If wallet is used, you need to set `HTTP_GW_WALLET_PASSPHRASE` variable to decrypt the wallet.
If no wallet provided, the gateway autogenerates a key pair it will use for NeoFS requests.
```
$ neofs-http-gw -p $NEOFS_NODE -w $WALLET_PATH --address $ACCOUNT_ADDRESS
```
@ -105,10 +105,10 @@ $ neofs-http-gw -p 192.168.130.72:8080 -w wallet.json --address NfgHwwTi3wHAS8aF
Gateway binds to `0.0.0.0:8082` by default and you can change that with
`--listen_address` option.
It can also provide TLS interface for its users, just specify paths to key and
It can also provide TLS interface for its users, just specify paths to the key and
certificate files via `--tls_key` and `--tls_certificate` parameters. Note
that using these options makes gateway TLS-only, if you need to serve both TLS
and plain text HTTP you either have to run two gateway instances or use some
that using these options makes gateway TLS-only. If you need to serve both TLS
and plain text HTTP, you either have to run two gateway instances or use some
external redirecting solution.
Example to bind to `192.168.130.130:443` and serve TLS there:
@ -131,7 +131,7 @@ and `HTTP_GW_WEB_WRITE_TIMEOUT=0`. Otherwise, HTTP Gateway will terminate
request with data stream after timeout.
`HTTP_GW_WEB_STREAM_REQUEST_BODY` environment variable can be used to disable
request body streaming (effectively it'll make gateway accept file completely
request body streaming (effectively it'll make the gateway accept the file completely
first and only then try sending it to NeoFS).
`HTTP_GW_WEB_MAX_REQUEST_BODY_SIZE` controls maximum request body size
@ -145,17 +145,17 @@ variable to control this behavior.
### Monitoring and metrics
Pprof and Prometheus are integrated into the gateway, but not enabled by
Pprof and Prometheus are integrated into the gateway, but they are not enabled by
default. To enable them use `--pprof` and `--metrics` flags or
`HTTP_GW_PPROF`/`HTTP_GW_METRICS` environment variables.
### Timeouts
You can tune gRPC interface parameters with `--connect_timeout` (for
connection to node) and `--request_timeout` (for request processing over
connection to a node) and `--request_timeout` (for request processing over
established connection) options.
gRPC-level checks allow gateway to detect dead peers, but it declares them
gRPC-level checks allow the gateway to detect dead peers, but it declares them
unhealthy at pool level once per `--rebalance_timer` interval, so check for it
if needed.
@ -194,16 +194,16 @@ supported.
Before uploading or downloading a file make sure you have a prepared container.
You can create it with instructions below.
Also in case of downloading you need to have a file inside a container.
Also ,in case of downloading, you need to have a file inside a container.
#### Create a container
You can create a container via [neofs-cli](https://github.com/nspcc-dev/neofs-node/releases):
You can create a container via [neofs-cli](https://github.com/nspcc-dev/neofs-node/releases):
```
$ neofs-cli -r $NEOFS_NODE -k $KEY container create --policy $POLICY --basic-acl $ACL
```
where `$KEY` can be a path to private key file (as raw bytes), a hex string or
(unencrypted) WIF string,
where `$KEY` can be a path to a private key file (as raw bytes), a hex string or
a (unencrypted) WIF string,
`$ACL` -- hex encoded basic ACL value or keywords 'private, 'public-read', 'public-read-write' and
`$POLICY` -- QL-encoded or JSON-encoded placement policy or path to file with it
@ -212,7 +212,7 @@ For example:
$ neofs-cli -r 192.168.130.72:8080 -k 6PYLKJhiSub5imt6WCVy6Quxtd9xu176omev1vWYovzkAQCTSQabAAQXii container create --policy "REP 3" --basic-acl public --await
```
If you launched nodes via [neofs-dev-env](https://github.com/nspcc-dev/neofs-dev-env)
If you have launched nodes via [neofs-dev-env](https://github.com/nspcc-dev/neofs-dev-env),
you can get the key value from `wallets/wallet.json` or write the path to
the file `wallets/wallet.key`.
@ -279,7 +279,7 @@ $ wget http://localhost:8082/get_by_attribute/88GdaZFTcYJn1dqiSECss8kKPmmun6d6Bf
$ wget http://localhost:8082/get_by_attribute/88GdaZFTcYJn1dqiSECss8kKPmmun6d6BfvC4zhwfLYM/FileName/cat%25jpeg # means 'cat%jpeg'
```
Some other user-defined attribute:
Some other user-defined attributes:
```
$ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/Ololo/100500
@ -332,10 +332,10 @@ set of reply headers generated using the following rules:
### Uploading
You can POST files to `/upload/$CID` path where `$CID` is container ID. The
You can POST files to `/upload/$CID` path where `$CID` is a container ID. The
request must contain multipart form with mandatory `filename` parameter. Only
one part in multipart form will be processed, so to upload another file just
issue new POST request.
issue a new POST request.
Example request:
@ -345,7 +345,7 @@ $ curl -F 'file=@cat.jpeg;filename=cat.jpeg' http://localhost:8082/upload/Dxhf4P
Chunked encoding is supported by the server (but check for request read
timeouts if you're planning some streaming). You can try streaming support
with large file piped through named FIFO pipe:
with a large file piped through named FIFO pipe:
```
$ mkfifo pipe
@ -383,7 +383,7 @@ which transforms to `X-Attribute-Neofs-Expiration-Epoch`. So you can provide exp
---
For successful uploads you get JSON data in reply body with container and
For successful uploads you get JSON data in reply body with a container and
object ID, like this:
```
{
@ -396,7 +396,7 @@ object ID, like this:
You can always upload files to public containers (open for anyone to put
objects into), but for restricted containers you need to explicitly allow PUT
operations for request signed with your HTTP Gateway keys.
operations for a request signed with your HTTP Gateway keys.
If your don't want to manage gateway's secret keys and adjust eACL rules when
gateway configuration changes (new gate, key rotation, etc) or you plan to use
@ -410,23 +410,23 @@ documentation for more details). There are two options to pass them to gateway:
credentials field
* "Bearer" cookie with base64-encoded token contents
For example you have a mobile application frontend with a backend part storing
data in NeoFS. When user authorizes in mobile app, the backend issues a NeoFS
Bearer token and provides it to the frontend. Then the mobile app may generate
For example, you have a mobile application frontend with a backend part storing
data in NeoFS. When a user authorizes in the mobile app, the backend issues a NeoFS
Bearer token and provides it to the frontend. Then, the mobile app may generate
some data and upload it via any available NeoFS HTTP Gateway by adding
the corresponding header to the upload request. Accessing the ACL protected data
works the same way.
##### Example
In order to generate bearer token, you need to know container owner key and
address of sender who will be do request to NeoFS (in our case it's gateway wallet address).
In order to generate a bearer token, you need to know the container owner key and
the address of the sender who will do the request to NeoFS (in our case, it's a gateway wallet address).
Suppose we have:
* **KxDgvEKzgSBPPfuVfw67oPQBSjidEiqTHURKSDL1R7yGaGYAeYnr** (container owner key)
* **NhVtreTTCoqsMQV5Wp55fqnriiUCpEaKm3** (token owner address)
* **BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K** (container id)
Firstly we need to encode container id and sender address to base64 (now it's base58).
Firstly, we need to encode the container id and the sender address to base64 (now it's base58).
So use **base58** and **base64** utils.
1. Encoding container id:
@ -441,7 +441,7 @@ $ echo 'NhVtreTTCoqsMQV5Wp55fqnriiUCpEaKm3' | base58 --decode | base64
# output: NezFK4ujidF+X7bB88uzREQzRQeAvdj3Gg==
```
Now we can form Bearer token (10000 is liftetime expiration in epoch) and save it to **bearer.json**:
Now, we can form a Bearer token (10000 is liftetime expiration in epoch) and save it to **bearer.json**:
```
{
"body": {
@ -468,17 +468,17 @@ Now we can form Bearer token (10000 is liftetime expiration in epoch) and save i
}
```
Then sign it with container owner key:
Next, sign it with the container owner key:
```
$ neofs-cli util sign bearer-token --from bearer.json --to signed.json -k KxDgvEKzgSBPPfuVfw67oPQBSjidEiqTHURKSDL1R7yGaGYAeYnr
```
Encoding to base64 to use via header:
Encoding to base64 to use via the header:
```
$ base64 -w 0 signed.json
# output: Ck4KKgoECAIQBhIiCiCZGdlbN7DPGPMg9rsWqV+p2XdMzUqknRiexewSFp8kmBIbChk17MUri6OJ0X5ftsHzy7NERDNFB4C92PcaGgMIkE4SZgohAxpsb7vfAso1F0X6hrm6WpRS14WsT3/Ct1SMoqRsT89KEkEEGxKi8GjKSf52YqhppgaOTQHbUsL3jn7SHLqS3ndAQ7NtAATnmRHleZw2V2xRRSRBQdjDC05KK83LhdSax72Fsw==
```
After that Bearer token can be used:
After that, the Bearer token can be used:
```
$ curl -F 'file=@cat.jpeg;filename=cat.jpeg' -H "Authorization: Bearer Ck4KKgoECAIQBhIiCiCZGdlbN7DPGPMg9rsWqV+p2XdMzUqknRiexewSFp8kmBIbChk17MUri6OJ0X5ftsHzy7NERDNFB4C92PcaGgMIkE4SZgohAxpsb7vfAso1F0X6hrm6WpRS14WsT3/Ct1SMoqRsT89KEkEEGxKi8GjKSf52YqhppgaOTQHbUsL3jn7SHLqS3ndAQ7NtAATnmRHleZw2V2xRRSRBQdjDC05KK83LhdSax72Fsw==" \
@ -500,7 +500,7 @@ For example:
$ neofs-cli --key KxDgvEKzgSBPPfuVfw67oPQBSjidEiqTHURKSDL1R7yGaGYAeYnr --basic-acl 0x0FFFCFFF -r 192.168.130.72:8080 container create --policy "REP 3" --await
```
To deny access to the container without a token, set the eACL rules:
To deny access to a container without a token, set the eACL rules:
```
$ neofs-cli --key KxDgvEKzgSBPPfuVfw67oPQBSjidEiqTHURKSDL1R7yGaGYAeYnr -r 192.168.130.72:8080 container set-eacl --table eacl.json --await --cid BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K
```

10
downloader/download.go
View file

@ -66,8 +66,8 @@ type readCloser struct {
io.Closer
}
// initializes io.Reader with limited size and detects Content-Type from it.
// Returns r's error directly. Also returns processed data.
// initializes io.Reader with the limited size and detects Content-Type from it.
// Returns r's error directly. Also returns the processed data.
func readContentType(maxSize uint64, rInit func(uint64) (io.Reader, error)) (string, []byte, error) {
if maxSize > sizeToDetectType {
maxSize = sizeToDetectType
@ -167,7 +167,7 @@ func (r request) receiveFile(clnt *pool.Pool, objectAddress *address.Address) {
return
}
// reset payload reader since part of the data has been read
// reset payload reader since a part of the data has been read
var headReader io.Reader = bytes.NewReader(payloadHead)
if err != io.EOF { // otherwise, we've already read full payload
@ -276,7 +276,7 @@ func (d *Downloader) DownloadByAddress(c *fasthttp.RequestCtx) {
d.byAddress(c, request.receiveFile)
}
// byAddress is wrapper for function (e.g. request.headObject, request.receiveFile) that
// byAddress is a wrapper for function (e.g. request.headObject, request.receiveFile) that
// prepares request and object address to it.
func (d *Downloader) byAddress(c *fasthttp.RequestCtx, f func(request, *pool.Pool, *address.Address)) {
var (
@ -300,7 +300,7 @@ func (d *Downloader) DownloadByAttribute(c *fasthttp.RequestCtx) {
d.byAttribute(c, request.receiveFile)
}
// byAttribute is wrapper similar to byAddress.
// byAttribute is a wrapper similar to byAddress.
func (d *Downloader) byAttribute(c *fasthttp.RequestCtx, f func(request, *pool.Pool, *address.Address)) {
var (
httpStatus = fasthttp.StatusBadRequest

2
misc.go
View file

@ -5,6 +5,6 @@ package main
const Prefix = "HTTP_GW"
var (
// Version is gateway version.
// Version is the gateway version.
Version = "dev"
)

8
tokens/bearer-token.go
View file

@ -26,7 +26,7 @@ const (
// return
// }
// BearerTokenFromHeader extracts bearer token from Authorization request header.
// BearerTokenFromHeader extracts a bearer token from Authorization request header.
func BearerTokenFromHeader(h *fasthttp.RequestHeader) []byte {
auth := h.Peek(fasthttp.HeaderAuthorization)
if auth == nil || !bytes.HasPrefix(auth, []byte(bearerTokenHdr)) {
@ -38,7 +38,7 @@ func BearerTokenFromHeader(h *fasthttp.RequestHeader) []byte {
return auth
}
// BearerTokenFromCookie extracts bearer token from cookies.
// BearerTokenFromCookie extracts a bearer token from cookies.
func BearerTokenFromCookie(h *fasthttp.RequestHeader) []byte {
auth := h.Cookie(bearerTokenHdr)
if len(auth) == 0 {
@ -48,7 +48,7 @@ func BearerTokenFromCookie(h *fasthttp.RequestHeader) []byte {
return auth
}
// StoreBearerToken extracts bearer token from header or cookie and stores
// StoreBearerToken extracts a bearer token from the header or cookie and stores
// it in the request context.
func StoreBearerToken(ctx *fasthttp.RequestCtx) error {
tkn, err := fetchBearerToken(ctx)
@ -60,7 +60,7 @@ func StoreBearerToken(ctx *fasthttp.RequestCtx) error {
return nil
}
// LoadBearerToken returns bearer token stored in context given (if it's
// LoadBearerToken returns a bearer token stored in the context given (if it's
// present there).
func LoadBearerToken(ctx context.Context) (*token.BearerToken, error) {
if tkn, ok := ctx.Value(bearerTokenKey).(*token.BearerToken); ok && tkn != nil {

8
uploader/filter.go
View file

@ -15,7 +15,7 @@ import (
var neofsAttributeHeaderPrefixes = [...][]byte{[]byte("Neofs-"), []byte("NEOFS-"), []byte("neofs-")}
func systemTranslator(key, prefix []byte) []byte {
// replace specified prefix with `__NEOFS__`
// replace the specified prefix with `__NEOFS__`
key = bytes.Replace(key, prefix, []byte(utils.SystemAttributePrefix), 1)
// replace `-` with `_`
@ -30,12 +30,12 @@ func filterHeaders(l *zap.Logger, header *fasthttp.RequestHeader) map[string]str
prefix := []byte(utils.UserAttributeHeaderPrefix)
header.VisitAll(func(key, val []byte) {
// checks that key and val not empty
// checks that the key and the val not empty
if len(key) == 0 || len(val) == 0 {
return
}
// checks that key has attribute prefix
// checks that the key has attribute prefix
if !bytes.HasPrefix(key, prefix) {
return
}
@ -51,7 +51,7 @@ func filterHeaders(l *zap.Logger, header *fasthttp.RequestHeader) map[string]str
}
}
// checks that attribute key not empty
// checks that the attribute key is not empty
if len(key) == 0 {
return
}