mbiryukova/frostfs-http-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-http-gw
Code Pull requests Activity

[#2] Update docs

Browse source 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
This commit is contained in:
Denis Kirillov 2022-12-16 11:37:22 +03:00 committed by Alex Vanin
parent 2d9eee81c2
commit a014fb96fc
5 changed files with 63 additions and 63 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.github/CODEOWNERS vendored
View file

@ -1 +1 @@
* @alexvanin @masterSplinter01 @KirillovDenis * @alexvanin @KirillovDenis

18
CONTRIBUTING.md
View file

@ -3,8 +3,8 @@
First, thank you for contributing! We love and encourage pull requests from First, thank you for contributing! We love and encourage pull requests from
everyone. Please follow the guidelines: everyone. Please follow the guidelines:
- Check the open [issues](https://github.com/nspcc-dev/neofs-http-gw/issues) and - Check the open [issues](https://github.com/TrueCloudLab/frostfs-http-gw/issues) and
[pull requests](https://github.com/nspcc-dev/neofs-http-gw/pulls) for existing [pull requests](https://github.com/TrueCloudLab/frostfs-http-gw/pulls) for existing
discussions. discussions.
- Open an issue first, to discuss a new feature or enhancement. - Open an issue first, to discuss a new feature or enhancement.
@ -23,24 +23,24 @@ everyone. Please follow the guidelines:
## Development Workflow ## Development Workflow
Start by forking the `neofs-http-gw` repository, make changes in a branch and then Start by forking the `frostfs-http-gw` repository, make changes in a branch and then
send a pull request. We encourage pull requests to discuss code changes. Here send a pull request. We encourage pull requests to discuss code changes. Here
are the steps in details: are the steps in details:
### Set up your GitHub Repository ### Set up your GitHub Repository
Fork [NeoFS HTTP Gateway Fork [FrostFS HTTP Gateway
upstream](https://github.com/nspcc-dev/neofs-http-gw/fork) source repository upstream](https://github.com/TrueCloudLab/frostfs-http-gw/fork) source repository
to your own personal repository. Copy the URL of your fork (you will need it for to your own personal repository. Copy the URL of your fork (you will need it for
the `git clone` command below). the `git clone` command below).
```sh ```sh
$ git clone https://github.com/nspcc-dev/neofs-http-gw $ git clone https://github.com/TrueCloudLab/frostfs-http-gw
``` ```
### Set up git remote as ``upstream`` ### Set up git remote as ``upstream``
```sh ```sh
$ cd neofs-http-gw $ cd frostfs-http-gw
$ git remote add upstream https://github.com/nspcc-dev/neofs-http-gw $ git remote add upstream https://github.com/TrueCloudLab/frostfs-http-gw
$ git fetch upstream $ git fetch upstream
$ git merge upstream/master $ git merge upstream/master
... ...
@ -107,7 +107,7 @@ contributors".
To sign your work, just add a line like this at the end of your commit message: To sign your work, just add a line like this at the end of your commit message:
``` ```
Signed-off-by: Samii Sakisaka <samii@nspcc.ru> Signed-off-by: Samii Sakisaka <samii@frostfs.info>
``` ```
This can be easily done with the `--signoff` option to `git commit`. This can be easily done with the `--signoff` option to `git commit`.

94
README.md
View file

@ -1,30 +1,30 @@
<p align="center"> <p align="center">
<img src="./.github/logo.svg" width="500px" alt="NeoFS"> <img src="./.github/logo.svg" width="500px" alt="FrostFS">
</p> </p>
<p align="center"> <p align="center">
<a href="https://fs.neo.org">NeoFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>. <a href="https://frostfs.info">FrostFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>.
</p> </p>
--- ---
[![Report](https://goreportcard.com/badge/github.com/nspcc-dev/neofs-http-gw)](https://goreportcard.com/report/github.com/nspcc-dev/neofs-http-gw) [![Report](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-http-gw)](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-http-gw)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/nspcc-dev/neofs-http-gw?sort=semver) ![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-http-gw?sort=semver)
![License](https://img.shields.io/github/license/nspcc-dev/neofs-http-gw.svg?style=popout) ![License](https://img.shields.io/github/license/TrueCloudLab/frostfs-http-gw.svg?style=popout)
# NeoFS HTTP Gateway # FrostFS HTTP Gateway
NeoFS HTTP Gateway bridges NeoFS internal protocol and HTTP standard. FrostFS HTTP Gateway bridges FrostFS internal protocol and HTTP standard.
- you can download one file per request from the NeoFS Network - you can download one file per request from the FrostFS Network
- you can upload one file per request into the NeoFS Network - you can upload one file per request into the FrostFS Network
See available routes in [specification](./docs/api.md). See available routes in [specification](./docs/api.md).
## Installation ## Installation
```go install github.com/nspcc-dev/neofs-http-gw``` ```go install github.com/TrueCloudLab/frostfs-http-gw```
Or you can call `make` to build it from the cloned repository (the binary will Or you can call `make` to build it from the cloned repository (the binary will
end up in `bin/neofs-http-gw`). To build neofs-http-gw binary in clean docker end up in `bin/frostfs-http-gw`). To build frostfs-http-gw binary in clean docker
environment, call `make docker/bin/neofs-http-gw`. environment, call `make docker/bin/frostfs-http-gw`.
Other notable make targets: Other notable make targets:
@ -44,26 +44,26 @@ latest stable release).
## Execution ## Execution
HTTP gateway itself is not a NeoFS node, so to access NeoFS it uses node's HTTP gateway itself is not a FrostFS node, so to access FrostFS it uses node's
gRPC interface and you need to provide some node that it will connect to. This gRPC interface and you need to provide some node that it will connect to. This
can be done either via `-p` parameter or via `HTTP_GW_PEERS_<N>_ADDRESS` and can be done either via `-p` parameter or via `HTTP_GW_PEERS_<N>_ADDRESS` and
`HTTP_GW_PEERS_<N>_WEIGHT` environment variables (the gate supports multiple `HTTP_GW_PEERS_<N>_WEIGHT` environment variables (the gate supports multiple
NeoFS nodes with weighted load balancing). FrostFS nodes with weighted load balancing).
If you launch HTTP gateway in bundle with [neofs-dev-env](https://github.com/nspcc-dev/neofs-dev-env), If you launch HTTP gateway in bundle with [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env),
you can get the IP address of the node in the output of `make hosts` command you can get the IP address of the node in the output of `make hosts` command
(with s0*.neofs.devenv name). (with s0*.neofs.devenv name).
These two commands are functionally equivalent, they run the gate with one These two commands are functionally equivalent, they run the gate with one
backend node (and otherwise default settings): backend node (and otherwise default settings):
``` ```
$ neofs-http-gw -p 192.168.130.72:8080 $ frostfs-http-gw -p 192.168.130.72:8080
$ HTTP_GW_PEERS_0_ADDRESS=192.168.130.72:8080 neofs-http-gw $ HTTP_GW_PEERS_0_ADDRESS=192.168.130.72:8080 frostfs-http-gw
``` ```
It's also possible to specify uri scheme (grpc or grpcs) when using `-p`: It's also possible to specify uri scheme (grpc or grpcs) when using `-p`:
``` ```
$ neofs-http-gw -p grpc://192.168.130.72:8080 $ frostfs-http-gw -p grpc://192.168.130.72:8080
$ HTTP_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 neofs-http-gw $ HTTP_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 frostfs-http-gw
``` ```
## Configuration ## Configuration
@ -74,11 +74,11 @@ environment variables (see [example](./config/config.env)), so they're not speci
### Nodes: weights and priorities ### Nodes: weights and priorities
You can specify multiple `-p` options to add more NeoFS nodes, this will make You can specify multiple `-p` options to add more FrostFS nodes, this will make
gateway spread requests equally among them (using weight 1 and priority 1 for every node): gateway spread requests equally among them (using weight 1 and priority 1 for every node):
``` ```
$ neofs-http-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080 $ frostfs-http-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080
``` ```
If you want some specific load distribution proportions, use weights and priorities: If you want some specific load distribution proportions, use weights and priorities:
@ -86,7 +86,7 @@ If you want some specific load distribution proportions, use weights and priorit
$ HTTP_GW_PEERS_0_ADDRESS=192.168.130.71:8080 HTTP_GW_PEERS_0_WEIGHT=1 HTTP_GW_PEERS_0_PRIORITY=1 \ $ HTTP_GW_PEERS_0_ADDRESS=192.168.130.71:8080 HTTP_GW_PEERS_0_WEIGHT=1 HTTP_GW_PEERS_0_PRIORITY=1 \
HTTP_GW_PEERS_1_ADDRESS=192.168.130.72:8080 HTTP_GW_PEERS_1_WEIGHT=9 HTTP_GW_PEERS_1_PRIORITY=2 \ HTTP_GW_PEERS_1_ADDRESS=192.168.130.72:8080 HTTP_GW_PEERS_1_WEIGHT=9 HTTP_GW_PEERS_1_PRIORITY=2 \
HTTP_GW_PEERS_2_ADDRESS=192.168.130.73:8080 HTTP_GW_PEERS_2_WEIGHT=1 HTTP_GW_PEERS_2_PRIORITY=2 \ HTTP_GW_PEERS_2_ADDRESS=192.168.130.73:8080 HTTP_GW_PEERS_2_WEIGHT=1 HTTP_GW_PEERS_2_PRIORITY=2 \
neofs-http-gw frostfs-http-gw
``` ```
This command will make gateway use 192.168.130.71 while it is healthy. Otherwise, it will make the gateway use This command will make gateway use 192.168.130.71 while it is healthy. Otherwise, it will make the gateway use
192.168.130.72 for 90% of requests and 192.168.130.73 for remaining 10%. 192.168.130.72 for 90% of requests and 192.168.130.73 for remaining 10%.
@ -94,13 +94,13 @@ This command will make gateway use 192.168.130.71 while it is healthy. Otherwise
### Keys ### Keys
You can provide a wallet via `--wallet` or `-w` flag. You can also specify the account address using `--address` You can provide a wallet via `--wallet` or `-w` flag. You can also specify the account address using `--address`
(if no address provided default one will be used). If wallet is used, you need to set `HTTP_GW_WALLET_PASSPHRASE` variable to decrypt the wallet. (if no address provided default one will be used). If wallet is used, you need to set `HTTP_GW_WALLET_PASSPHRASE` variable to decrypt the wallet.
If no wallet provided, the gateway autogenerates a key pair it will use for NeoFS requests. If no wallet provided, the gateway autogenerates a key pair it will use for FrostFS requests.
``` ```
$ neofs-http-gw -p $NEOFS_NODE -w $WALLET_PATH --address $ACCOUNT_ADDRESS $ frostfs-http-gw -p $NEOFS_NODE -w $WALLET_PATH --address $ACCOUNT_ADDRESS
``` ```
Example: Example:
``` ```
$ neofs-http-gw -p 192.168.130.72:8080 -w wallet.json --address NfgHwwTi3wHAS8aFAN243C5vGbkYDpqLHP $ frostfs-http-gw -p 192.168.130.72:8080 -w wallet.json --address NfgHwwTi3wHAS8aFAN243C5vGbkYDpqLHP
``` ```
### Binding and TLS ### Binding and TLS
@ -116,7 +116,7 @@ external redirecting solution.
Example to bind to `192.168.130.130:443` and serve TLS there: Example to bind to `192.168.130.130:443` and serve TLS there:
``` ```
$ neofs-http-gw -p 192.168.130.72:8080 --listen_address 192.168.130.130:443 \ $ frostfs-http-gw -p 192.168.130.72:8080 --listen_address 192.168.130.130:443 \
--tls_key=key.pem --tls_certificate=cert.pem --tls_key=key.pem --tls_certificate=cert.pem
``` ```
@ -134,12 +134,12 @@ request with data stream after timeout.
`HTTP_GW_WEB_STREAM_REQUEST_BODY` environment variable can be used to disable `HTTP_GW_WEB_STREAM_REQUEST_BODY` environment variable can be used to disable
request body streaming (effectively it'll make the gateway accept the file completely request body streaming (effectively it'll make the gateway accept the file completely
first and only then try sending it to NeoFS). first and only then try sending it to FrostFS).
`HTTP_GW_WEB_MAX_REQUEST_BODY_SIZE` controls maximum request body size `HTTP_GW_WEB_MAX_REQUEST_BODY_SIZE` controls maximum request body size
limiting uploads to files slightly lower than this limit. limiting uploads to files slightly lower than this limit.
### NeoFS parameters ### FrostFS parameters
Gateway can automatically set timestamps for uploaded files based on local Gateway can automatically set timestamps for uploaded files based on local
time source, use `HTTP_GW_UPLOAD_HEADER_USE_DEFAULT_TIMESTAMP` environment time source, use `HTTP_GW_UPLOAD_HEADER_USE_DEFAULT_TIMESTAMP` environment
@ -177,7 +177,7 @@ HTTP_GW_LOGGER_LEVEL=debug
Configuration file is optional and can be used instead of environment variables/other parameters. Configuration file is optional and can be used instead of environment variables/other parameters.
It can be specified with `--config` parameter: It can be specified with `--config` parameter:
``` ```
$ neofs-http-gw --config your-config.yaml $ frostfs-http-gw --config your-config.yaml
``` ```
See [config](./config/config.yaml) and [defaults](./docs/gate-configuration.md) for example. See [config](./config/config.yaml) and [defaults](./docs/gate-configuration.md) for example.
@ -185,7 +185,7 @@ See [config](./config/config.yaml) and [defaults](./docs/gate-configuration.md)
## HTTP API provided ## HTTP API provided
This gateway intentionally provides limited feature set and doesn't try to This gateway intentionally provides limited feature set and doesn't try to
substitute (or completely wrap) regular gRPC NeoFS interface. You can download substitute (or completely wrap) regular gRPC FrostFS interface. You can download
and upload objects with it, but deleting, searching, managing ACLs, creating and upload objects with it, but deleting, searching, managing ACLs, creating
containers and other activities are not supported and not planned to be containers and other activities are not supported and not planned to be
supported. supported.
@ -211,7 +211,7 @@ resolve_order:
- nns - nns
``` ```
2. Make sure your container is registered in NNS contract. If you use [neofs-dev-env](https://github.com/nspcc-dev/neofs-dev-env) 2. Make sure your container is registered in NNS contract. If you use [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env)
you can check if your container (e.g. with `container-name` name) is registered in NNS: you can check if your container (e.g. with `container-name` name) is registered in NNS:
```shell ```shell
@ -238,9 +238,9 @@ $ curl http://localhost:8082/get_by_attribute/container-name/FileName/object-nam
#### Create a container #### Create a container
You can create a container via [neofs-cli](https://github.com/nspcc-dev/neofs-node/releases): You can create a container via [frostfs-cli](https://github.com/TrueCloudLab/frostfs-node/releases):
``` ```
$ neofs-cli -r $NEOFS_NODE -w $WALLET container create --policy $POLICY --basic-acl $ACL $ frostfs-cli -r $NEOFS_NODE -w $WALLET container create --policy $POLICY --basic-acl $ACL
``` ```
where `$WALLET` is a path to user wallet, where `$WALLET` is a path to user wallet,
`$ACL` -- hex encoded basic ACL value or keywords 'private, 'public-read', 'public-read-write' and `$ACL` -- hex encoded basic ACL value or keywords 'private, 'public-read', 'public-read-write' and
@ -248,18 +248,18 @@ where `$WALLET` is a path to user wallet,
For example: For example:
``` ```
$ neofs-cli -r 192.168.130.72:8080 -w ./wallet.json container create --policy "REP 3" --basic-acl public --await $ frostfs-cli -r 192.168.130.72:8080 -w ./wallet.json container create --policy "REP 3" --basic-acl public --await
``` ```
If you have launched nodes via [neofs-dev-env](https://github.com/nspcc-dev/neofs-dev-env), If you have launched nodes via [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env),
you can get the key value from `wallets/wallet.json` or write the path to you can get the key value from `wallets/wallet.json` or write the path to
the file `wallets/wallet.key`. the file `wallets/wallet.key`.
#### Prepare a file in a container #### Prepare a file in a container
To create a file via [neofs-cli](https://github.com/nspcc-dev/neofs-node/releases), run a command below: To create a file via [frostfs-cli](https://github.com/TrueCloudLab/frostfs-node/releases), run a command below:
``` ```
$ neofs-cli -r $NEOFS_NODE -k $KEY object put --file $FILENAME --cid $CID $ frostfs-cli -r $NEOFS_NODE -k $KEY object put --file $FILENAME --cid $CID
``` ```
where where
`$KEY` -- the key, please read the information [above](#create-a-container), `$KEY` -- the key, please read the information [above](#create-a-container),
@ -267,7 +267,7 @@ where
For example: For example:
``` ```
$ neofs-cli -r 192.168.130.72:8080 -w ./wallet.json object put --file cat.png --cid Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ --attributes img_type=cat,my_attr=cute $ frostfs-cli -r 192.168.130.72:8080 -w ./wallet.json object put --file cat.png --cid Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ --attributes img_type=cat,my_attr=cute
``` ```
@ -372,7 +372,7 @@ set of reply headers generated using the following rules:
* `x-container-id` contains container ID * `x-container-id` contains container ID
* `x-object-id` contains object ID * `x-object-id` contains object ID
* `x-owner-id` contains owner address * `x-owner-id` contains owner address
* all the other NeoFS attributes are converted to `X-Attribute-*` headers (but only * all the other FrostFS attributes are converted to `X-Attribute-*` headers (but only
if they can be safely represented in HTTP header), for example `FileName` if they can be safely represented in HTTP header), for example `FileName`
attribute becomes `X-Attribute-FileName` header attribute becomes `X-Attribute-FileName` header
@ -452,25 +452,25 @@ operations for a request signed with your HTTP Gateway keys.
If your don't want to manage gateway's secret keys and adjust eACL rules when If your don't want to manage gateway's secret keys and adjust eACL rules when
gateway configuration changes (new gate, key rotation, etc) or you plan to use gateway configuration changes (new gate, key rotation, etc) or you plan to use
public services, there is an option to let your application backend (or you) to public services, there is an option to let your application backend (or you) to
issue Bearer Tokens ans pass them from the client via gate down to NeoFS level issue Bearer Tokens ans pass them from the client via gate down to FrostFS level
to grant access. to grant access.
NeoFS Bearer Token basically is a container owner-signed ACL data (refer to NeoFS FrostFS Bearer Token basically is a container owner-signed ACL data (refer to FrostFS
documentation for more details). There are two options to pass them to gateway: documentation for more details). There are two options to pass them to gateway:
* "Authorization" header with "Bearer" type and base64-encoded token in * "Authorization" header with "Bearer" type and base64-encoded token in
credentials field credentials field
* "Bearer" cookie with base64-encoded token contents * "Bearer" cookie with base64-encoded token contents
For example, you have a mobile application frontend with a backend part storing For example, you have a mobile application frontend with a backend part storing
data in NeoFS. When a user authorizes in the mobile app, the backend issues a NeoFS data in FrostFS. When a user authorizes in the mobile app, the backend issues a FrostFS
Bearer token and provides it to the frontend. Then, the mobile app may generate Bearer token and provides it to the frontend. Then, the mobile app may generate
some data and upload it via any available NeoFS HTTP Gateway by adding some data and upload it via any available FrostFS HTTP Gateway by adding
the corresponding header to the upload request. Accessing the ACL protected data the corresponding header to the upload request. Accessing the ACL protected data
works the same way. works the same way.
##### Example ##### Example
In order to generate a bearer token, you need to know the container owner key and In order to generate a bearer token, you need to know the container owner key and
the address of the sender who will do the request to NeoFS (in our case, it's a gateway wallet address). the address of the sender who will do the request to FrostFS (in our case, it's a gateway wallet address).
Suppose we have: Suppose we have:
* **KxDgvEKzgSBPPfuVfw67oPQBSjidEiqTHURKSDL1R7yGaGYAeYnr** (container owner key) * **KxDgvEKzgSBPPfuVfw67oPQBSjidEiqTHURKSDL1R7yGaGYAeYnr** (container owner key)
@ -521,7 +521,7 @@ Now, we can form a Bearer token (10000 is liftetime expiration in epoch) and sav
Next, sign it with the container owner key: Next, sign it with the container owner key:
``` ```
$ neofs-cli util sign bearer-token --from bearer.json --to signed.json -w ./wallet.json $ frostfs-cli util sign bearer-token --from bearer.json --to signed.json -w ./wallet.json
``` ```
Encoding to base64 to use via the header: Encoding to base64 to use via the header:
``` ```
@ -548,12 +548,12 @@ For the token to work correctly, you need to create a container with a basic ACL
For example: For example:
``` ```
$ neofs-cli -w ./wallet.json --basic-acl 0x0FFFCFFF -r 192.168.130.72:8080 container create --policy "REP 3" --await $ frostfs-cli -w ./wallet.json --basic-acl 0x0FFFCFFF -r 192.168.130.72:8080 container create --policy "REP 3" --await
``` ```
To deny access to a container without a token, set the eACL rules: To deny access to a container without a token, set the eACL rules:
``` ```
$ neofs-cli -w ./wallet.json -r 192.168.130.72:8080 container set-eacl --table eacl.json --await --cid BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K $ frostfs-cli -w ./wallet.json -r 192.168.130.72:8080 container set-eacl --table eacl.json --await --cid BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K
``` ```
File **eacl.json**: File **eacl.json**:

2
docs/api.md
View file

@ -50,7 +50,7 @@ Route: `/upload/{cid}`
#### POST #### POST
Upload file as object with attributes to NeoFS. Upload file as object with attributes to FrostFS.
##### Request ##### Request

10
docs/gate-configuration.md
View file

@ -1,6 +1,6 @@
# NeoFS HTTP Gateway configuration file # FrostFS HTTP Gateway configuration file
This section contains detailed NeoFS HTTP Gateway configuration file description This section contains detailed FrostFS HTTP Gateway configuration file description
including default config values and some tips to set up configurable values. including default config values and some tips to set up configurable values.
There are some custom types used for brevity: There are some custom types used for brevity:
@ -23,19 +23,19 @@ $ kill -s SIGHUP <app_pid>
Example: Example:
```shell ```shell
$ ./bin/neofs-http-gw --config config.yaml &> http.log & $ ./bin/frostfs-http-gw --config config.yaml &> http.log &
[1] 998346 [1] 998346
$ cat http.log $ cat http.log
# ... # ...
2022-10-03T09:37:25.826+0300 info neofs-http-gw/app.go:332 starting application {"app_name": "neofs-http-gw", "version": "v0.24.0"} 2022-10-03T09:37:25.826+0300 info frostfs-http-gw/app.go:332 starting application {"app_name": "frostfs-http-gw", "version": "v0.24.0"}
# ... # ...
$ kill -s SIGHUP 998346 $ kill -s SIGHUP 998346
$ cat http.log $ cat http.log
# ... # ...
2022-10-03T09:38:16.205+0300 info neofs-http-gw/app.go:470 SIGHUP config reload completed 2022-10-03T09:38:16.205+0300 info frostfs-http-gw/app.go:470 SIGHUP config reload completed
``` ```
# Structure # Structure