2021-04-30 16:45:03 +00:00
< p align = "center" >
2022-12-16 08:37:22 +00:00
< img src = "./.github/logo.svg" width = "500px" alt = "FrostFS" >
2021-04-30 16:45:03 +00:00
< / p >
< p align = "center" >
2022-12-16 08:37:22 +00:00
< a href = "https://frostfs.info" > FrostFS< / a > is a decentralized distributed object storage integrated with the < a href = "https://neo.org" > NEO Blockchain< / a > .
2021-04-30 16:45:03 +00:00
< / p >
---
2022-12-16 08:37:22 +00:00
[![Report ](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-http-gw )](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-http-gw)
![GitHub release (latest SemVer) ](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-http-gw?sort=semver )
![License ](https://img.shields.io/github/license/TrueCloudLab/frostfs-http-gw.svg?style=popout )
2021-04-30 16:45:03 +00:00
2022-12-16 08:37:22 +00:00
# FrostFS HTTP Gateway
2019-11-07 17:29:29 +00:00
2022-12-16 08:37:22 +00:00
FrostFS HTTP Gateway bridges FrostFS internal protocol and HTTP standard.
- you can download one file per request from the FrostFS Network
- you can upload one file per request into the FrostFS Network
2019-11-07 17:29:29 +00:00
2022-10-27 11:18:48 +00:00
See available routes in [specification ](./docs/api.md ).
2021-04-29 21:15:04 +00:00
## Installation
2022-12-16 08:37:22 +00:00
```go install github.com/TrueCloudLab/frostfs-http-gw```
2021-04-29 21:15:04 +00:00
Or you can call `make` to build it from the cloned repository (the binary will
2022-12-16 08:37:22 +00:00
end up in `bin/frostfs-http-gw` ). To build frostfs-http-gw binary in clean docker
environment, call `make docker/bin/frostfs-http-gw` .
2021-04-29 21:15:04 +00:00
2022-06-07 16:58:33 +00:00
Other notable make targets:
2021-02-05 14:13:37 +00:00
```
2021-04-09 15:29:18 +00:00
dep Check and ensure dependencies
image Build clean docker image
2021-04-29 21:15:04 +00:00
dirty-image Build dirty docker image with host-built binaries
2022-06-07 17:10:23 +00:00
fmt Format the code
2021-04-09 15:29:18 +00:00
lint Run linters
version Show current version
2021-02-05 14:13:37 +00:00
```
2021-05-04 15:19:11 +00:00
Or you can also use a [Docker
2022-12-16 08:45:27 +00:00
image](https://hub.docker.com/r/truecloudlab/frostfs-http-gw) provided for the released
2022-04-21 08:35:57 +00:00
(and occasionally unreleased) versions of the gateway (`:latest` points to the
2021-05-04 15:19:11 +00:00
latest stable release).
2021-04-29 21:15:04 +00:00
## Execution
2019-11-07 17:29:29 +00:00
2022-12-16 08:37:22 +00:00
HTTP gateway itself is not a FrostFS node, so to access FrostFS it uses node's
2021-04-29 21:15:04 +00:00
gRPC interface and you need to provide some node that it will connect to. This
can be done either via `-p` parameter or via `HTTP_GW_PEERS_<N>_ADDRESS` and
`HTTP_GW_PEERS_<N>_WEIGHT` environment variables (the gate supports multiple
2022-12-16 08:37:22 +00:00
FrostFS nodes with weighted load balancing).
2021-04-29 21:15:04 +00:00
2022-12-16 08:37:22 +00:00
If you launch HTTP gateway in bundle with [frostfs-dev-env ](https://github.com/TrueCloudLab/frostfs-dev-env ),
2022-04-21 08:35:57 +00:00
you can get the IP address of the node in the output of `make hosts` command
2022-12-20 11:01:50 +00:00
(with s0*.frostfs.devenv name).
2021-05-17 14:30:16 +00:00
2021-04-29 21:15:04 +00:00
These two commands are functionally equivalent, they run the gate with one
backend node (and otherwise default settings):
```
2022-12-16 08:37:22 +00:00
$ frostfs-http-gw -p 192.168.130.72:8080
$ HTTP_GW_PEERS_0_ADDRESS=192.168.130.72:8080 frostfs-http-gw
2021-04-29 21:15:04 +00:00
```
2021-06-09 16:26:27 +00:00
It's also possible to specify uri scheme (grpc or grpcs) when using `-p` :
```
2022-12-16 08:37:22 +00:00
$ frostfs-http-gw -p grpc://192.168.130.72:8080
$ HTTP_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 frostfs-http-gw
2021-06-09 16:26:27 +00:00
```
2021-04-29 21:15:04 +00:00
2021-05-04 15:19:11 +00:00
## Configuration
2021-04-29 21:15:04 +00:00
In general, everything available as CLI parameter can also be specified via
2022-04-08 09:01:57 +00:00
environment variables (see [example ](./config/config.env )), so they're not specifically mentioned in most cases
2021-06-28 10:24:46 +00:00
(see `--help` also). If you prefer a config file you can use it in yaml format.
2021-04-29 21:15:04 +00:00
2021-12-07 11:57:16 +00:00
### Nodes: weights and priorities
2021-04-29 21:15:04 +00:00
2022-12-16 08:37:22 +00:00
You can specify multiple `-p` options to add more FrostFS nodes, this will make
2021-12-07 11:57:16 +00:00
gateway spread requests equally among them (using weight 1 and priority 1 for every node):
2021-04-29 21:15:04 +00:00
```
2022-12-16 08:37:22 +00:00
$ frostfs-http-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080
2021-04-29 21:15:04 +00:00
```
2021-12-07 11:57:16 +00:00
If you want some specific load distribution proportions, use weights and priorities:
2021-04-29 21:15:04 +00:00
```
2021-12-07 11:57:16 +00:00
$ HTTP_GW_PEERS_0_ADDRESS=192.168.130.71:8080 HTTP_GW_PEERS_0_WEIGHT=1 HTTP_GW_PEERS_0_PRIORITY=1 \
HTTP_GW_PEERS_1_ADDRESS=192.168.130.72:8080 HTTP_GW_PEERS_1_WEIGHT=9 HTTP_GW_PEERS_1_PRIORITY=2 \
HTTP_GW_PEERS_2_ADDRESS=192.168.130.73:8080 HTTP_GW_PEERS_2_WEIGHT=1 HTTP_GW_PEERS_2_PRIORITY=2 \
2022-12-16 08:37:22 +00:00
frostfs-http-gw
2021-04-29 21:15:04 +00:00
```
2021-12-07 11:57:16 +00:00
This command will make gateway use 192.168.130.71 while it is healthy. Otherwise, it will make the gateway use
192.168.130.72 for 90% of requests and 192.168.130.73 for remaining 10%.
2021-04-29 21:15:04 +00:00
2021-05-04 15:19:11 +00:00
### Keys
2022-04-21 08:35:57 +00:00
You can provide a wallet via `--wallet` or `-w` flag. You can also specify the account address using `--address`
(if no address provided default one will be used). If wallet is used, you need to set `HTTP_GW_WALLET_PASSPHRASE` variable to decrypt the wallet.
2022-12-16 08:37:22 +00:00
If no wallet provided, the gateway autogenerates a key pair it will use for FrostFS requests.
2021-05-17 14:30:16 +00:00
```
2022-12-20 11:01:50 +00:00
$ frostfs-http-gw -p $FROSTFS_NODE -w $WALLET_PATH --address $ACCOUNT_ADDRESS
2021-05-17 14:30:16 +00:00
```
2021-06-22 11:53:59 +00:00
Example:
2021-04-29 21:15:04 +00:00
```
2022-12-16 08:37:22 +00:00
$ frostfs-http-gw -p 192.168.130.72:8080 -w wallet.json --address NfgHwwTi3wHAS8aFAN243C5vGbkYDpqLHP
2021-04-29 21:15:04 +00:00
```
2019-11-07 17:29:29 +00:00
2021-05-04 15:19:11 +00:00
### Binding and TLS
2021-04-29 21:15:04 +00:00
2022-07-19 14:06:17 +00:00
You can make the gateway listen on specific address using the `--listen_address` option.
2021-04-29 21:15:04 +00:00
2022-04-21 08:35:57 +00:00
It can also provide TLS interface for its users, just specify paths to the key and
2021-04-29 21:15:04 +00:00
certificate files via `--tls_key` and `--tls_certificate` parameters. Note
2022-04-21 08:35:57 +00:00
that using these options makes gateway TLS-only. If you need to serve both TLS
and plain text HTTP, you either have to run two gateway instances or use some
2021-04-29 21:15:04 +00:00
external redirecting solution.
Example to bind to `192.168.130.130:443` and serve TLS there:
```
2022-12-16 08:37:22 +00:00
$ frostfs-http-gw -p 192.168.130.72:8080 --listen_address 192.168.130.130:443 \
2021-04-29 21:15:04 +00:00
--tls_key=key.pem --tls_certificate=cert.pem
```
2021-05-04 15:19:11 +00:00
### HTTP parameters
2021-04-29 21:15:04 +00:00
You can tune HTTP read and write buffer sizes as well as timeouts with
`HTTP_GW_WEB_READ_BUFFER_SIZE` , `HTTP_GW_WEB_READ_TIMEOUT` ,
`HTTP_GW_WEB_WRITE_BUFFER_SIZE` and `HTTP_GW_WEB_WRITE_TIMEOUT` environment
variables.
2022-04-21 15:53:17 +00:00
**Note:** to allow upload and download of big data streams, disable read
and write timeouts correspondingly. To do that, set `HTTP_GW_WEB_READ_TIMEOUT=0`
and `HTTP_GW_WEB_WRITE_TIMEOUT=0` . Otherwise, HTTP Gateway will terminate
request with data stream after timeout.
2021-04-29 21:15:04 +00:00
`HTTP_GW_WEB_STREAM_REQUEST_BODY` environment variable can be used to disable
2022-04-21 08:35:57 +00:00
request body streaming (effectively it'll make the gateway accept the file completely
2022-12-16 08:37:22 +00:00
first and only then try sending it to FrostFS).
2021-04-29 21:15:04 +00:00
`HTTP_GW_WEB_MAX_REQUEST_BODY_SIZE` controls maximum request body size
limiting uploads to files slightly lower than this limit.
2022-12-16 08:37:22 +00:00
### FrostFS parameters
2021-04-29 21:15:04 +00:00
Gateway can automatically set timestamps for uploaded files based on local
time source, use `HTTP_GW_UPLOAD_HEADER_USE_DEFAULT_TIMESTAMP` environment
variable to control this behavior.
2021-05-04 15:19:11 +00:00
### Monitoring and metrics
2021-04-29 21:15:04 +00:00
2022-07-19 14:06:17 +00:00
Pprof and Prometheus are integrated into the gateway. To enable them use `--pprof` and `--metrics` flags or
2021-04-29 21:15:04 +00:00
`HTTP_GW_PPROF` /`HTTP_GW_METRICS` environment variables.
2021-05-04 15:19:11 +00:00
### Timeouts
2021-04-29 21:15:04 +00:00
You can tune gRPC interface parameters with `--connect_timeout` (for
2022-04-21 08:35:57 +00:00
connection to a node) and `--request_timeout` (for request processing over
2021-06-09 16:26:27 +00:00
established connection) options.
2021-04-29 21:15:04 +00:00
2022-04-21 08:35:57 +00:00
gRPC-level checks allow the gateway to detect dead peers, but it declares them
2021-04-29 21:15:04 +00:00
unhealthy at pool level once per `--rebalance_timer` interval, so check for it
if needed.
All timing options accept values with suffixes, so "15s" is 15 seconds and
"2m" is 2 minutes.
2021-10-29 13:11:34 +00:00
### Zip streaming
The gateway supports downloading files by common prefix (like dir) in zip format. You can enable compression
using config or `HTTP_GW_ZIP_COMPRESSION=true` environment variable.
2021-05-04 15:19:11 +00:00
### Logging
2022-07-19 14:06:17 +00:00
You can specify logging level using variable:
2021-04-29 21:15:04 +00:00
```
2022-04-08 09:01:57 +00:00
HTTP_GW_LOGGER_LEVEL=debug
2021-03-30 22:46:33 +00:00
```
2021-04-29 21:15:04 +00:00
2021-06-28 10:24:46 +00:00
### Yaml file
Configuration file is optional and can be used instead of environment variables/other parameters.
It can be specified with `--config` parameter:
```
2022-12-16 08:37:22 +00:00
$ frostfs-http-gw --config your-config.yaml
2021-06-28 10:24:46 +00:00
```
2022-07-19 14:06:17 +00:00
See [config ](./config/config.yaml ) and [defaults ](./docs/gate-configuration.md ) for example.
2021-06-28 10:24:46 +00:00
2021-04-29 21:15:04 +00:00
## HTTP API provided
This gateway intentionally provides limited feature set and doesn't try to
2022-12-16 08:37:22 +00:00
substitute (or completely wrap) regular gRPC FrostFS interface. You can download
2021-04-29 21:15:04 +00:00
and upload objects with it, but deleting, searching, managing ACLs, creating
2021-05-04 15:19:11 +00:00
containers and other activities are not supported and not planned to be
2021-04-29 21:15:04 +00:00
supported.
2021-05-17 14:30:16 +00:00
### Preparation
Before uploading or downloading a file make sure you have a prepared container.
You can create it with instructions below.
2022-08-12 15:30:23 +00:00
Also, in case of downloading, you need to have a file inside a container.
### NNS
In all download/upload routes you can use container name instead of its id (`$CID`).
Steps to start using name resolving:
1. Enable NNS resolving in config (`rpc_endpoint` must be a valid neo rpc node, see [configs ](./config ) for other examples):
```yaml
2022-12-20 11:01:50 +00:00
rpc_endpoint: http://morph-chain.frostfs.devenv:30333
2022-08-12 15:30:23 +00:00
resolve_order:
- nns
```
2022-12-16 08:37:22 +00:00
2. Make sure your container is registered in NNS contract. If you use [frostfs-dev-env ](https://github.com/TrueCloudLab/frostfs-dev-env )
2022-08-12 15:30:23 +00:00
you can check if your container (e.g. with `container-name` name) is registered in NNS:
```shell
$ curl -s --data '{"id":1,"jsonrpc":"2.0","method":"getcontractstate","params":[1]}' \
2022-12-20 11:01:50 +00:00
http://morph-chain.frostfs.devenv:30333 | jq -r '.result.hash'
2022-08-12 15:30:23 +00:00
0x8e6c3cd4b976b28e84a3788f6ea9e2676c15d667
$ docker exec -it morph_chain neo-go \
contract testinvokefunction \
2022-12-20 11:01:50 +00:00
-r http://morph-chain.frostfs.devenv:30333 0x8e6c3cd4b976b28e84a3788f6ea9e2676c15d667 \
2022-08-12 15:30:23 +00:00
resolve string:container-name.container int:16 \
| jq -r '.stack[0].value | if type=="array" then .[0].value else . end' \
| base64 -d & & echo
7f3vvkw4iTiS5ZZbu5BQXEmJtETWbi3uUjLNaSs29xrL
```
3. Use container name instead of its `$CID` . For example:
```shell
$ curl http://localhost:8082/get_by_attribute/container-name/FileName/object-name
```
2021-05-17 14:30:16 +00:00
#### Create a container
2022-12-16 08:37:22 +00:00
You can create a container via [frostfs-cli ](https://github.com/TrueCloudLab/frostfs-node/releases ):
2021-05-17 14:30:16 +00:00
```
2022-12-20 11:01:50 +00:00
$ frostfs-cli -r $FROSTFS_NODE -w $WALLET container create --policy $POLICY --basic-acl $ACL
2021-05-17 14:30:16 +00:00
```
2022-08-12 15:30:23 +00:00
where `$WALLET` is a path to user wallet,
2021-06-04 07:38:01 +00:00
`$ACL` -- hex encoded basic ACL value or keywords 'private, 'public-read', 'public-read-write' and
2021-05-17 14:30:16 +00:00
`$POLICY` -- QL-encoded or JSON-encoded placement policy or path to file with it
For example:
```
2022-12-16 08:37:22 +00:00
$ frostfs-cli -r 192.168.130.72:8080 -w ./wallet.json container create --policy "REP 3" --basic-acl public --await
2021-05-17 14:30:16 +00:00
```
2022-12-16 08:37:22 +00:00
If you have launched nodes via [frostfs-dev-env ](https://github.com/TrueCloudLab/frostfs-dev-env ),
2021-05-17 14:30:16 +00:00
you can get the key value from `wallets/wallet.json` or write the path to
the file `wallets/wallet.key` .
#### Prepare a file in a container
2022-12-16 08:37:22 +00:00
To create a file via [frostfs-cli ](https://github.com/TrueCloudLab/frostfs-node/releases ), run a command below:
2021-05-17 14:30:16 +00:00
```
2022-12-20 11:01:50 +00:00
$ frostfs-cli -r $FROSTFS_NODE -k $KEY object put --file $FILENAME --cid $CID
2021-05-17 14:30:16 +00:00
```
where
`$KEY` -- the key, please read the information [above ](#create-a-container ),
`$CID` -- container ID.
For example:
```
2022-12-16 08:37:22 +00:00
$ frostfs-cli -r 192.168.130.72:8080 -w ./wallet.json object put --file cat.png --cid Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ --attributes img_type=cat,my_attr=cute
2021-05-17 14:30:16 +00:00
```
2021-04-29 21:15:04 +00:00
### Downloading
#### Requests
2021-07-15 07:57:22 +00:00
The following requests support GET/HEAD methods.
2021-05-17 14:30:16 +00:00
##### By IDs
Basic downloading involves container ID and object ID and is done via GET
2022-08-12 15:30:23 +00:00
requests to `/get/$CID/$OID` path, where `$CID` is a container ID or its name if NNS is enabled,
2021-05-17 14:30:16 +00:00
`$OID` is an object's (i.e. your file's) ID.
2022-08-12 15:30:23 +00:00
For example:
2021-04-29 21:15:04 +00:00
2022-08-12 15:30:23 +00:00
```shell
2021-04-29 21:15:04 +00:00
$ wget http://localhost:8082/get/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocY
```
2022-08-12 15:30:23 +00:00
or if container has a name:
```shell
$ wget http://localhost:8082/get/container-name/2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocY
```
2021-05-17 14:30:16 +00:00
##### By attributes
2021-04-29 21:15:04 +00:00
There is also more complex interface provided for attribute-based downloads,
it's usually used to retrieve files by their names, but any other attribute
can be used as well. The generic syntax for it looks like this:
```/get_by_attribute/$CID/$ATTRIBUTE_NAME/$ATTRIBUTE_VALUE```
2021-05-17 14:30:16 +00:00
where
2022-08-12 15:30:23 +00:00
`$CID` is a container ID or its name if NNS is enabled,
2021-05-17 14:30:16 +00:00
`$ATTRIBUTE_NAME` is the name of the attribute we want to use,
`$ATTRIBUTE_VALUE` is the value of this attribute that the target object should have.
2021-04-29 21:15:04 +00:00
2022-10-26 14:20:25 +00:00
**NB!** The attribute key and value should be url encoded, i.e., if you want to download an object with the attribute value
`a cat` , the value in the request must be `a+cat` . In the same way with the attribute key. If you don't escape such values
everything can still work (for example you can use `d@ta` without encoding) but it's HIGHLY RECOMMENDED to encode all your attributes.
2022-01-24 14:04:34 +00:00
2021-04-29 21:15:04 +00:00
If multiple objects have specified attribute with specified value, then the
first one of them is returned (and you can't get others via this interface).
Example for file name attribute:
```
$ wget http://localhost:8082/get_by_attribute/88GdaZFTcYJn1dqiSECss8kKPmmun6d6BfvC4zhwfLYM/FileName/cat.jpeg
```
2022-01-24 14:04:34 +00:00
Or when the filename includes special symbols:
```
$ wget http://localhost:8082/get_by_attribute/88GdaZFTcYJn1dqiSECss8kKPmmun6d6BfvC4zhwfLYM/FileName/cat+jpeg # means 'cat jpeg'
$ wget http://localhost:8082/get_by_attribute/88GdaZFTcYJn1dqiSECss8kKPmmun6d6BfvC4zhwfLYM/FileName/cat%25jpeg # means 'cat%jpeg'
```
2021-04-29 21:15:04 +00:00
2022-04-21 08:35:57 +00:00
Some other user-defined attributes:
2021-04-29 21:15:04 +00:00
```
$ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/Ololo/100500
```
2022-01-24 14:04:34 +00:00
Or when the attribute includes special symbols:
```
$ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/Olo%2Blo/100500 # means Olo+lo
```
2021-04-29 21:15:04 +00:00
An optional `download=true` argument for `Content-Disposition` management is
also supported (more on that below):
```
$ wget http://localhost:8082/get/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocY?download=true
```
2021-10-29 13:11:34 +00:00
##### Zip
You can download some dir (files with the same prefix) in zip (it will be compressed if config contains appropriate param):
```
$ wget http://localhost:8082/zip/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/common/prefix
```
2022-11-07 12:31:37 +00:00
**Note:** the objects must have a valid `FilePath` attribute (it should not contain trailing `/` ),
otherwise they will not be in the zip archive. You can upload file with this attribute using `curl` :
2022-04-19 12:09:00 +00:00
```
$ curl -F 'file=@cat.jpeg;filename=cat.jpeg' -H 'X-Attribute-FilePath: common/prefix/cat.jpeg' http://localhost:8082/upload/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ
```
2021-04-29 21:15:04 +00:00
#### Replies
2021-07-15 07:57:22 +00:00
You get object contents in the reply body (if GET method was used), but at the same time you also get a
2021-04-29 21:15:04 +00:00
set of reply headers generated using the following rules:
* `Content-Length` is set to the length of the object
* `Content-Type` is autodetected dynamically by gateway
* `Content-Disposition` is `inline` for regular requests and `attachment` for
requests with `download=true` argument, `filename` is also added if there
is `FileName` attribute set for this object
* `Last-Modified` header is set to `Timestamp` attribute value if it's
present for the object
* `x-container-id` contains container ID
* `x-object-id` contains object ID
* `x-owner-id` contains owner address
2022-12-16 08:37:22 +00:00
* all the other FrostFS attributes are converted to `X-Attribute-*` headers (but only
2021-04-29 21:15:04 +00:00
if they can be safely represented in HTTP header), for example `FileName`
2021-04-30 09:55:51 +00:00
attribute becomes `X-Attribute-FileName` header
2021-04-29 21:15:04 +00:00
2022-11-14 07:47:40 +00:00
##### Caching strategy
HTTP Gateway doesn't control caching (doesn't anything with the `Cache-Control` header). Caching strategy strictly
depends on application use case. So it should be carefully done by proxy server.
2021-04-29 21:15:04 +00:00
### Uploading
2022-08-12 15:30:23 +00:00
You can POST files to `/upload/$CID` path where `$CID` is a container ID or its name if NNS is enabled. The
2021-04-29 21:15:04 +00:00
request must contain multipart form with mandatory `filename` parameter. Only
one part in multipart form will be processed, so to upload another file just
2022-04-21 08:35:57 +00:00
issue a new POST request.
2021-04-29 21:15:04 +00:00
Example request:
```
$ curl -F 'file=@cat.jpeg;filename=cat.jpeg' http://localhost:8082/upload/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ
```
Chunked encoding is supported by the server (but check for request read
timeouts if you're planning some streaming). You can try streaming support
2022-04-21 08:35:57 +00:00
with a large file piped through named FIFO pipe:
2021-04-29 21:15:04 +00:00
```
$ mkfifo pipe
$ cat video.mp4 > pipe &
$ curl --no-buffer -F 'file=@pipe;filename=catvideo.mp4' http://localhost:8082/upload/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ
```
You can also add some attributes to your file using the following rules:
* all "X-Attribute-*" headers get converted to object attributes with
"X-Attribute-" prefix stripped, that is if you add "X-Attribute-Ololo:
100500" header to your request the resulting object will get "Ololo:
100500" attribute
2021-09-08 06:15:42 +00:00
* "X-Attribute-NEOFS-*" headers are special
(`-NEOFS-` part can also be `-neofs-` or`-Neofs-`), they're used to set internal
2021-04-29 21:15:04 +00:00
NeoFS attributes starting with `__NEOFS__` prefix, for these attributes all
dashes get converted to underscores and all letters are capitalized. For
example, you can use "X-Attribute-NEOFS-Expiration-Epoch" header to set
`__NEOFS__EXPIRATION_EPOCH` attribute
* `FileName` attribute is set from multipart's `filename` if not set
explicitly via `X-Attribute-FileName` header
* `Timestamp` attribute can be set using gateway local time if using
HTTP_GW_UPLOAD_HEADER_USE_DEFAULT_TIMESTAMP option and if request doesn't
provide `X-Attribute-Timestamp` header of its own
2021-11-26 12:44:07 +00:00
---
**NOTE**
There are some reserved headers type of `X-Attribute-NEOFS-*` (headers are arranged in descending order of priority):
1. `X-Attribute-Neofs-Expiration-Epoch: 100`
2. `X-Attribute-Neofs-Expiration-Duration: 24h30m`
3. `X-Attribute-Neofs-Expiration-Timestamp: 1637574797`
4. `X-Attribute-Neofs-Expiration-RFC3339: 2021-11-22T09:55:49Z`
which transforms to `X-Attribute-Neofs-Expiration-Epoch` . So you can provide expiration any convenient way.
---
2022-04-21 08:35:57 +00:00
For successful uploads you get JSON data in reply body with a container and
2021-04-29 21:15:04 +00:00
object ID, like this:
```
{
"object_id": "9ANhbry2ryjJY1NZbcjryJMRXG5uGNKd73kD3V1sVFsX",
"container_id": "Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ"
}
```
2021-04-30 10:41:45 +00:00
#### Authentication
You can always upload files to public containers (open for anyone to put
2021-05-04 15:19:11 +00:00
objects into), but for restricted containers you need to explicitly allow PUT
2022-04-21 08:35:57 +00:00
operations for a request signed with your HTTP Gateway keys.
2021-05-04 15:19:11 +00:00
If your don't want to manage gateway's secret keys and adjust eACL rules when
gateway configuration changes (new gate, key rotation, etc) or you plan to use
public services, there is an option to let your application backend (or you) to
2022-12-16 08:37:22 +00:00
issue Bearer Tokens ans pass them from the client via gate down to FrostFS level
2021-05-04 15:19:11 +00:00
to grant access.
2022-12-16 08:37:22 +00:00
FrostFS Bearer Token basically is a container owner-signed ACL data (refer to FrostFS
2021-05-04 15:19:11 +00:00
documentation for more details). There are two options to pass them to gateway:
2021-04-30 10:41:45 +00:00
* "Authorization" header with "Bearer" type and base64-encoded token in
credentials field
2021-06-03 12:18:27 +00:00
* "Bearer" cookie with base64-encoded token contents
2021-04-30 10:41:45 +00:00
2022-04-21 08:35:57 +00:00
For example, you have a mobile application frontend with a backend part storing
2022-12-16 08:37:22 +00:00
data in FrostFS. When a user authorizes in the mobile app, the backend issues a FrostFS
2022-04-21 08:35:57 +00:00
Bearer token and provides it to the frontend. Then, the mobile app may generate
2022-12-16 08:37:22 +00:00
some data and upload it via any available FrostFS HTTP Gateway by adding
2021-05-04 15:19:11 +00:00
the corresponding header to the upload request. Accessing the ACL protected data
works the same way.
2021-06-03 14:11:54 +00:00
##### Example
2022-04-21 08:35:57 +00:00
In order to generate a bearer token, you need to know the container owner key and
2022-12-16 08:37:22 +00:00
the address of the sender who will do the request to FrostFS (in our case, it's a gateway wallet address).
2021-06-03 14:11:54 +00:00
Suppose we have:
* **KxDgvEKzgSBPPfuVfw67oPQBSjidEiqTHURKSDL1R7yGaGYAeYnr** (container owner key)
* **NhVtreTTCoqsMQV5Wp55fqnriiUCpEaKm3** (token owner address)
* **BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K** (container id)
2022-04-21 08:35:57 +00:00
Firstly, we need to encode the container id and the sender address to base64 (now it's base58).
2021-06-03 14:11:54 +00:00
So use **base58** and **base64** utils.
1. Encoding container id:
```
$ echo 'BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K' | base58 --decode | base64
# output: mRnZWzewzxjzIPa7Fqlfqdl3TM1KpJ0YnsXsEhafJJg=
```
2. Encoding token owner id:
```
$ echo 'NhVtreTTCoqsMQV5Wp55fqnriiUCpEaKm3' | base58 --decode | base64
# output: NezFK4ujidF+X7bB88uzREQzRQeAvdj3Gg==
```
2022-04-21 08:35:57 +00:00
Now, we can form a Bearer token (10000 is liftetime expiration in epoch) and save it to **bearer.json** :
2021-06-03 14:11:54 +00:00
```
{
"body": {
"eaclTable": {
"version": {
"major": 0,
"minor": 0
},
"containerID": {
"value": "mRnZWzewzxjzIPa7Fqlfqdl3TM1KpJ0YnsXsEhafJJg="
},
"records": []
},
"ownerID": {
"value": "NezFK4ujidF+X7bB88uzREQzRQeAvdj3Gg=="
},
"lifetime": {
"exp": "10000",
"nbf": "0",
"iat": "0"
}
},
"signature": null
}
```
2022-04-21 08:35:57 +00:00
Next, sign it with the container owner key:
2021-06-03 14:11:54 +00:00
```
2022-12-16 08:37:22 +00:00
$ frostfs-cli util sign bearer-token --from bearer.json --to signed.json -w ./wallet.json
2021-06-03 14:11:54 +00:00
```
2022-04-21 08:35:57 +00:00
Encoding to base64 to use via the header:
2021-06-03 14:11:54 +00:00
```
$ base64 -w 0 signed.json
# output: Ck4KKgoECAIQBhIiCiCZGdlbN7DPGPMg9rsWqV+p2XdMzUqknRiexewSFp8kmBIbChk17MUri6OJ0X5ftsHzy7NERDNFB4C92PcaGgMIkE4SZgohAxpsb7vfAso1F0X6hrm6WpRS14WsT3/Ct1SMoqRsT89KEkEEGxKi8GjKSf52YqhppgaOTQHbUsL3jn7SHLqS3ndAQ7NtAATnmRHleZw2V2xRRSRBQdjDC05KK83LhdSax72Fsw==
```
2022-04-21 08:35:57 +00:00
After that, the Bearer token can be used:
2021-06-03 14:11:54 +00:00
```
$ curl -F 'file=@cat.jpeg;filename=cat.jpeg' -H "Authorization: Bearer Ck4KKgoECAIQBhIiCiCZGdlbN7DPGPMg9rsWqV+p2XdMzUqknRiexewSFp8kmBIbChk17MUri6OJ0X5ftsHzy7NERDNFB4C92PcaGgMIkE4SZgohAxpsb7vfAso1F0X6hrm6WpRS14WsT3/Ct1SMoqRsT89KEkEEGxKi8GjKSf52YqhppgaOTQHbUsL3jn7SHLqS3ndAQ7NtAATnmRHleZw2V2xRRSRBQdjDC05KK83LhdSax72Fsw==" \
http://localhost:8082/upload/BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K
# output:
# {
# "object_id": "DhfES9nVrFksxGDD2jQLunGADfrXExxNwqXbDafyBn9X",
# "container_id": "BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K"
# }
```
2021-06-07 12:37:03 +00:00
##### Note
For the token to work correctly, you need to create a container with a basic ACL that:
1. Allow PUT operation to others
2. Doesn't set "final" bit
For example:
```
2022-12-16 08:37:22 +00:00
$ frostfs-cli -w ./wallet.json --basic-acl 0x0FFFCFFF -r 192.168.130.72:8080 container create --policy "REP 3" --await
2021-06-07 12:37:03 +00:00
```
2022-04-21 08:35:57 +00:00
To deny access to a container without a token, set the eACL rules:
2021-06-07 12:37:03 +00:00
```
2022-12-16 08:37:22 +00:00
$ frostfs-cli -w ./wallet.json -r 192.168.130.72:8080 container set-eacl --table eacl.json --await --cid BJeErH9MWmf52VsR1mLWKkgF3pRm3FkubYxM7TZkBP4K
2021-06-07 12:37:03 +00:00
```
File **eacl.json** :
```
{
"version": {
"major": 0,
"minor": 0
},
"containerID": {
"value": "mRnZWzewzxjzIPa7Fqlfqdl3TM1KpJ0YnsXsEhafJJg="
},
"records": [
{
"operation": "PUT",
"action": "DENY",
"filters": [],
"targets": [
{
"role": "OTHERS",
"keys": []
}
]
}
]
}
```
2021-04-29 21:15:04 +00:00
### Metrics and Pprof
2022-08-12 15:30:23 +00:00
If enabled, Prometheus metrics are available at `localhost:8084` endpoint
and Pprof at `localhost:8083/debug/pprof` by default. Host and port can be configured.
See [configuration ](./docs/gate-configuration.md ).
2022-07-14 08:49:22 +00:00
## Credits
Please see [CREDITS ](CREDITS.md ) for details.