forked from TrueCloudLab/frostfs-s3-gw
Alex Vanin
parent
96dff367db
commit
09c6e22b84
8 changed files with 89 additions and 89 deletions
2
.github/CODEOWNERS
vendored
2
.github/CODEOWNERS
vendored
|
|
@ -1 +1 @@
|
|||
* @alexvanin @masterSplinter01 @KirillovDenis
|
||||
* @alexvanin @KirillovDenis
|
||||
|
|
|
|||
|
|
@ -3,8 +3,8 @@
|
|||
First, thank you for contributing! We love and encourage pull requests from
|
||||
everyone. Please follow the guidelines:
|
||||
|
||||
- Check the open [issues](https://github.com/nspcc-dev/neofs-s3-gw/issues) and
|
||||
[pull requests](https://github.com/nspcc-dev/neofs-s3-gw/pulls) for existing
|
||||
- Check the open [issues](https://github.com/TrueCloudLab/frostfs-s3-gw/issues) and
|
||||
[pull requests](https://github.com/TrueCloudLab/frostfs-s3-gw/pulls) for existing
|
||||
discussions.
|
||||
|
||||
- Open an issue first, to discuss a new feature or enhancement.
|
||||
|
|
@ -23,24 +23,24 @@ everyone. Please follow the guidelines:
|
|||
|
||||
## Development Workflow
|
||||
|
||||
Start by forking the `neofs-s3-gw` repository, make changes in a branch and then
|
||||
Start by forking the `frostfs-s3-gw` repository, make changes in a branch and then
|
||||
send a pull request. We encourage pull requests to discuss code changes. Here
|
||||
are the steps in details:
|
||||
|
||||
### Set up your GitHub Repository
|
||||
Fork [NeoFS S3 Gateway
|
||||
upstream](https://github.com/nspcc-dev/neofs-s3-gw/fork) source repository
|
||||
Fork [FrostFS S3 Gateway
|
||||
upstream](https://github.com/TrueCloudLab/frostfs-s3-gw/fork) source repository
|
||||
to your own personal repository. Copy the URL of your fork (you will need it for
|
||||
the `git clone` command below).
|
||||
|
||||
```sh
|
||||
$ git clone https://github.com/nspcc-dev/neofs-s3-gw
|
||||
$ git clone https://github.com/TrueCloudLab/frostfs-s3-gw
|
||||
```
|
||||
|
||||
### Set up git remote as ``upstream``
|
||||
```sh
|
||||
$ cd neofs-s3-gw
|
||||
$ git remote add upstream https://github.com/nspcc-dev/neofs-s3-gw
|
||||
$ cd frostfs-s3-gw
|
||||
$ git remote add upstream https://github.com/TrueCloudLab/frostfs-s3-gw
|
||||
$ git fetch upstream
|
||||
$ git merge upstream/master
|
||||
...
|
||||
|
|
@ -107,7 +107,7 @@ contributors".
|
|||
To sign your work, just add a line like this at the end of your commit message:
|
||||
|
||||
```
|
||||
Signed-off-by: Samii Sakisaka <samii@nspcc.ru>
|
||||
Signed-off-by: Samii Sakisaka <samii@frostfs.info>
|
||||
```
|
||||
|
||||
This can be easily done with the `--signoff` option to `git commit`.
|
||||
|
|
|
|||
30
README.md
30
README.md
|
|
@ -1,13 +1,13 @@
|
|||
# NeoFS S3 Gateway
|
||||
# FrostFS S3 Gateway
|
||||
|
||||
NeoFS S3 gateway provides API compatible with Amazon S3 cloud storage service.
|
||||
FrostFS S3 gateway provides API compatible with Amazon S3 cloud storage service.
|
||||
|
||||
## Installation
|
||||
|
||||
```go get -u github.com/nspcc-dev/neofs-s3-gw```
|
||||
```go get -u github.com/TrueCloudLab/frostfs-s3-gw```
|
||||
|
||||
Or you can call `make` to build it from the cloned repository (the binary will
|
||||
end up in `bin/neofs-s3-gw` with authmate helper in `bin/neofs-s3-authmate`).
|
||||
end up in `bin/frostfs-s3-gw` with authmate helper in `bin/frostfs-s3-authmate`).
|
||||
To build binaries in clean docker environment, call `make docker/all`.
|
||||
|
||||
Other notable make targets:
|
||||
|
|
@ -22,36 +22,36 @@ version Show current version
|
|||
```
|
||||
|
||||
Or you can also use a [Docker
|
||||
image](https://hub.docker.com/r/nspccdev/neofs-s3-gw) provided for released
|
||||
image](https://hub.docker.com/r/nspccdev/frostfs-s3-gw) provided for released
|
||||
(and occasionally unreleased) versions of gateway (`:latest` points to the
|
||||
latest stable release).
|
||||
|
||||
## Execution
|
||||
|
||||
Minimalistic S3 gateway setup needs:
|
||||
* NeoFS node(s) address (S3 gateway itself is not a NeoFS node)
|
||||
* FrostFS node(s) address (S3 gateway itself is not a FrostFS node)
|
||||
Passed via `-p` parameter or via `S3_GW_PEERS_<N>_ADDRESS` and
|
||||
`S3_GW_PEERS_<N>_WEIGHT` environment variables (gateway supports multiple
|
||||
NeoFS nodes with weighted load balancing).
|
||||
* a wallet used to fetch key and communicate with NeoFS nodes
|
||||
FrostFS nodes with weighted load balancing).
|
||||
* a wallet used to fetch key and communicate with FrostFS nodes
|
||||
Passed via `--wallet` parameter or `S3_GW_WALLET_PATH` environment variable.
|
||||
|
||||
These two commands are functionally equivalent, they run the gate with one
|
||||
backend node, some keys and otherwise default settings:
|
||||
```
|
||||
$ neofs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json
|
||||
$ frostfs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json
|
||||
|
||||
$ S3_GW_PEERS_0_ADDRESS=192.168.130.72:8080 \
|
||||
S3_GW_WALLET=wallet.json \
|
||||
neofs-s3-gw
|
||||
frostfs-s3-gw
|
||||
```
|
||||
It's also possible to specify uri scheme (grpc or grpcs) when using `-p` or environment variables:
|
||||
```
|
||||
$ neofs-s3-gw -p grpc://192.168.130.72:8080 --wallet wallet.json
|
||||
$ frostfs-s3-gw -p grpc://192.168.130.72:8080 --wallet wallet.json
|
||||
|
||||
$ S3_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 \
|
||||
S3_GW_WALLET=wallet.json \
|
||||
neofs-s3-gw
|
||||
frostfs-s3-gw
|
||||
```
|
||||
|
||||
## Domains
|
||||
|
|
@ -60,7 +60,7 @@ By default, s3-gw enable only `path-style access`.
|
|||
To be able to use both: `virtual-hosted-style` and `path-style` access you must configure `listen_domains`:
|
||||
|
||||
```shell
|
||||
$ neofs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json --listen_domains your.first.domain --listen_domains your.second.domain
|
||||
$ frostfs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json --listen_domains your.first.domain --listen_domains your.second.domain
|
||||
```
|
||||
|
||||
So now you can use (e.g. `HeadBucket`. Make sure DNS is properly configured):
|
||||
|
|
@ -84,8 +84,8 @@ Also, you can configure domains using `.env` variables or `yaml` file.
|
|||
## Documentation
|
||||
|
||||
- [Configuration](./docs/configuration.md)
|
||||
- [NeoFS S3 AuthMate](./docs/authmate.md)
|
||||
- [NeoFS Tree service](./docs/tree_service.md)
|
||||
- [FrostFS S3 AuthMate](./docs/authmate.md)
|
||||
- [FrostFS Tree service](./docs/tree_service.md)
|
||||
- [AWS CLI basic usage](./docs/aws_cli.md)
|
||||
- [AWS S3 API compatibility](./docs/aws_s3_compat.md)
|
||||
- [AWS S3 Compatibility test results](./docs/s3_test_results.md)
|
||||
|
|
|
|||
|
|
@ -1,18 +1,18 @@
|
|||
# NeoFS S3 AuthMate
|
||||
# FrostFS S3 AuthMate
|
||||
|
||||
Authmate is a tool to create gateway AWS credentials. AWS users
|
||||
are authenticated with access key IDs and secrets, while NeoFS users are
|
||||
are authenticated with access key IDs and secrets, while FrostFS users are
|
||||
authenticated with key pairs. To complicate things further, we have S3 gateway
|
||||
that usually acts on behalf of some user, but the user doesn't necessarily want to
|
||||
give their keys to the gateway.
|
||||
|
||||
To solve this, we use NeoFS bearer tokens that are signed by the owner (NeoFS
|
||||
"user") and that can implement any kind of policy for NeoFS requests allowed
|
||||
To solve this, we use FrostFS bearer tokens that are signed by the owner (FrostFS
|
||||
"user") and that can implement any kind of policy for FrostFS requests allowed
|
||||
to use this token. However, tokens can't be used as AWS credentials directly. Thus,
|
||||
they're stored on NeoFS as regular objects, and an access key ID is just an
|
||||
they're stored on FrostFS as regular objects, and an access key ID is just an
|
||||
address of this object while a secret is generated randomly.
|
||||
|
||||
Tokens are not stored on NeoFS in plaintext, they're encrypted with a set of
|
||||
Tokens are not stored on FrostFS in plaintext, they're encrypted with a set of
|
||||
gateway keys. So, in order for a gateway to be able to successfully extract bearer
|
||||
token, the object needs to be stored in a container available for the gateway
|
||||
to read, and it needs to be encrypted with this gateway's key (among others
|
||||
|
|
@ -83,7 +83,7 @@ NhLQpDnerpviUWDF77j5qyjFgavCmasJ4p (simple signature contract):
|
|||
## Issuance of a secret
|
||||
|
||||
To issue a secret means to create Bearer and, optionally, Session tokens and
|
||||
put them as an object into a container on the NeoFS network.
|
||||
put them as an object into a container on the FrostFS network.
|
||||
|
||||
### CLI parameters
|
||||
|
||||
|
|
@ -91,7 +91,7 @@ put them as an object into a container on the NeoFS network.
|
|||
* `--wallet` is a path to a wallet `.json` file. You can provide a passphrase to decrypt
|
||||
a wallet via environment variable `AUTHMATE_WALLET_PASSPHRASE`, or you will be asked to enter a passphrase
|
||||
interactively. You can also specify an account address to use from a wallet using the `--address` parameter.
|
||||
* `--peer` is an address of a NeoFS peer to connect to
|
||||
* `--peer` is an address of a FrostFS peer to connect to
|
||||
* `--gate-public-key` is a public `secp256r1` 33-byte short key of a gate (use flags repeatedly for multiple gates). The tokens are encrypted
|
||||
by a set of gateway keys, so you need to pass them as well.
|
||||
|
||||
|
|
@ -105,7 +105,7 @@ You can issue a secret using the parameters above only. The tool will
|
|||
|
||||
E.g.:
|
||||
```shell
|
||||
$ neofs-s3-authmate issue-secret --wallet wallet.json \
|
||||
$ frostfs-s3-authmate issue-secret --wallet wallet.json \
|
||||
--peer 192.168.130.71:8080 \
|
||||
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf\
|
||||
--gate-public-key 0317585fa8274f7afdf1fc5f2a2e7bece549d5175c4e5182e37924f30229aef967
|
||||
|
|
@ -122,7 +122,7 @@ $ neofs-s3-authmate issue-secret --wallet wallet.json \
|
|||
|
||||
`access_key_id` and `secret_access_key` are AWS credentials that you can use with any S3 client.
|
||||
|
||||
`access_key_id` consists of Base58 encoded containerID(cid) and objectID(oid) stored on the NeoFS network and containing
|
||||
`access_key_id` consists of Base58 encoded containerID(cid) and objectID(oid) stored on the FrostFS network and containing
|
||||
the secret. Format of `access_key_id`: `%cid0%oid`, where 0(zero) is a delimiter.
|
||||
|
||||
**Optional parameters:**
|
||||
|
|
@ -141,7 +141,7 @@ Creation of bearer tokens is mandatory.
|
|||
|
||||
Rules for a bearer token can be set via parameter `--bearer-rules` (json-string and file path allowed):
|
||||
```shell
|
||||
$ neofs-s3-authmate issue-secret --wallet wallet.json \
|
||||
$ frostfs-s3-authmate issue-secret --wallet wallet.json \
|
||||
--peer 192.168.130.71:8080 \
|
||||
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \
|
||||
--bearer-rules bearer-rules.json
|
||||
|
|
@ -195,7 +195,7 @@ If bearer rules are not set, a token will be auto-generated with a value:
|
|||
With a session token, there are 3 options:
|
||||
1. append `--session-tokens` parameter with your custom rules in json format (as a string or file path). E.g.:
|
||||
```shell
|
||||
$ neofs-s3-authmate issue-secret --wallet wallet.json \
|
||||
$ frostfs-s3-authmate issue-secret --wallet wallet.json \
|
||||
--peer 192.168.130.71:8080 \
|
||||
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \
|
||||
--session-tokens session.json
|
||||
|
|
@ -224,7 +224,7 @@ If `containerID` is `null` or omitted, then session token rule will be applied
|
|||
to all containers. Otherwise, specify `containerID` value in human-redabale
|
||||
format (base58 encoded string).
|
||||
|
||||
> **_NB!_** To create buckets in NeoFS it's necessary to have session tokens with `PUT` and `SETEACL` permissions, that's why
|
||||
> **_NB!_** To create buckets in FrostFS it's necessary to have session tokens with `PUT` and `SETEACL` permissions, that's why
|
||||
the authmate creates a `SETEACL` session token automatically in case when a user specified the token rule with `PUT` and
|
||||
forgot about the rule with `SETEACL`.
|
||||
|
||||
|
|
@ -235,7 +235,7 @@ in example above)
|
|||
### Containers policy
|
||||
|
||||
Rules for mapping of `LocationConstraint` ([aws spec](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html#API_CreateBucket_RequestBody))
|
||||
to `PlacementPolicy` ([neofs spec](https://github.com/nspcc-dev/neofs-spec/blob/master/01-arch/02-policy.md))
|
||||
to `PlacementPolicy` ([frostfs spec](https://github.com/TrueCloudLab/frostfs-spec/blob/master/01-arch/02-policy.md))
|
||||
can be set via parameter `--container-policy` (json-string and file path allowed):
|
||||
```json
|
||||
{
|
||||
|
|
@ -248,12 +248,12 @@ can be set via parameter `--container-policy` (json-string and file path allowed
|
|||
## Obtainment of a secret access key
|
||||
|
||||
You can get a secret access key associated with an access key ID by obtaining a
|
||||
secret stored on the NeoFS network. Here is an example of providing one password (for `wallet.json`) via env variable
|
||||
secret stored on the FrostFS network. Here is an example of providing one password (for `wallet.json`) via env variable
|
||||
and the other (for `gate-wallet.json`) interactively:
|
||||
|
||||
```shell
|
||||
$ AUTHMATE_WALLET_PASSPHRASE=some-pwd \
|
||||
neofs-s3-authmate obtain-secret --wallet wallet.json \
|
||||
frostfs-s3-authmate obtain-secret --wallet wallet.json \
|
||||
--peer 192.168.130.71:8080 \
|
||||
--gate-wallet gate-wallet.json \
|
||||
--access-key-id 5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM
|
||||
|
|
@ -272,7 +272,7 @@ using AWS credentials from `~/.aws/credentials` (you can specify profile using t
|
|||
with the following command:
|
||||
|
||||
```shell
|
||||
$ neofs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \
|
||||
$ frostfs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \
|
||||
--method get --bucket presigned --object obj --lifetime 30s
|
||||
|
||||
{
|
||||
|
|
@ -283,7 +283,7 @@ $ neofs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \
|
|||
You can also provide credential explicitly:
|
||||
|
||||
```shell
|
||||
$ neofs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \
|
||||
$ frostfs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \
|
||||
--method put --bucket presigned --object obj --lifetime 12h \
|
||||
--region ru --aws-secret-access-key c2d65ef2980f03f4f495bdebedeeae760496697880d61d106bb9a4e5cd2e0607 \
|
||||
--aws-access-key-id ETaA2CadPcA7bAkLsML2PbTudXY8uRt2PDjCCwkvRv9s0FDCxWDXYc1SA1vKv8KbyCNsLY2AmAjJ92Vz5rgvsFCy
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
|
||||
To configure basic settings that the AWS CLI uses to interact with the Gateway, follow the steps below:
|
||||
|
||||
1. issue a secret with neofs-s3-authmate tool (see [NeoFS S3 Authmate](./authmate.md))
|
||||
1. issue a secret with frostfs-s3-authmate tool (see [FrostFS S3 Authmate](./authmate.md))
|
||||
2. execute the command
|
||||
```
|
||||
$ aws configure
|
||||
|
|
@ -28,7 +28,7 @@ Default output format [none]: json
|
|||
|
||||
#### Obtainment of a list of buckets
|
||||
|
||||
To view the list of the buckets in the NeoFS node, to which the gateway is connected, enter the following command:
|
||||
To view the list of the buckets in the FrostFS node, to which the gateway is connected, enter the following command:
|
||||
```
|
||||
$ aws s3 ls
|
||||
```
|
||||
|
|
@ -72,15 +72,15 @@ $ aws s3api list-objects --bucket %BUCKET_NAME
|
|||
|
||||
#### Upload of a file
|
||||
|
||||
To upload a file into a bucket in the NeoFS network, run the following command:
|
||||
To upload a file into a bucket in the FrostFS network, run the following command:
|
||||
```
|
||||
$ aws s3api put-object --bucket %BUCKET_NAME --key %OBJECT_KEY --body %FILEPATH
|
||||
```
|
||||
where %OBJECT_KEY is the filepath of an object in NeoFS
|
||||
where %OBJECT_KEY is the filepath of an object in FrostFS
|
||||
|
||||
#### Upload of a dir
|
||||
|
||||
To upload a dir into a bucket in the NeoFS network, run the following command:
|
||||
To upload a dir into a bucket in the FrostFS network, run the following command:
|
||||
|
||||
```
|
||||
$ aws s3 sync %DIRPATH s3://%BUCKET_NAME
|
||||
|
|
@ -88,7 +88,7 @@ $ aws s3 sync %DIRPATH s3://%BUCKET_NAME
|
|||
|
||||
#### Download of a file
|
||||
|
||||
To download a file from a bucket in the NeoFS Network, execute:
|
||||
To download a file from a bucket in the FrostFS Network, execute:
|
||||
```
|
||||
$ aws s3api get-object --bucket %BUCKET_NAME --key %OBJECT_KEY %OUTFILE
|
||||
```
|
||||
|
|
|
|||
|
|
@ -19,20 +19,20 @@ basic configuration can be completed with CLI parameters only.
|
|||
3. [Binding and TLS](#listening-on-address-and-TLS)
|
||||
4. [RPC endpoint and resolving of bucket names](#rpc-endpoint-and-resolving-of-bucket-names)
|
||||
5. [Processing of requests](#processing-of-requests)
|
||||
6. [Connection to NeoFS](#connection-to-NeoFS)
|
||||
6. [Connection to FrostFS](#connection-to-FrostFS)
|
||||
7. [Monitoring and metrics](#monitoring-and-metrics)
|
||||
2. [YAML file and environment variables](#yaml-file-and-environment-variables)
|
||||
1. [Configuration file](#neofs-s3-gateway-configuration-file)
|
||||
1. [Configuration file](#frostfs-s3-gateway-configuration-file)
|
||||
|
||||
## CLI parameters
|
||||
|
||||
### Nodes and weights
|
||||
|
||||
You can specify multiple `-p` options to add more NeoFS nodes; this will make
|
||||
You can specify multiple `-p` options to add more FrostFS nodes; this will make
|
||||
a gateway spread requests equally among them (using weight 1 for every node):
|
||||
|
||||
```shell
|
||||
$ neofs-s3-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080
|
||||
$ frostfs-s3-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080
|
||||
```
|
||||
|
||||
If you want some specific load distribution proportions, use weights and priorities, they
|
||||
|
|
@ -58,7 +58,7 @@ Example to bind to `192.168.130.130:443` and serve TLS there (keys and nodes are
|
|||
omitted):
|
||||
|
||||
```shell
|
||||
$ neofs-s3-gw --listen_address 192.168.130.130:443 \
|
||||
$ frostfs-s3-gw --listen_address 192.168.130.130:443 \
|
||||
--tls.key_file=key.pem --tls.cert_file=cert.pem
|
||||
```
|
||||
|
||||
|
|
@ -70,7 +70,7 @@ To set RPC endpoint specify a value of parameter `-r` or `--rpc_endpoint`. The p
|
|||
parameter's `--resolve_order` value contains `nns`.
|
||||
|
||||
```shell
|
||||
$ neofs-s3-gw --rpc_endpoint http://morph-chain.neofs.devenv:30333/ --resolve_order nns,dns
|
||||
$ frostfs-s3-gw --rpc_endpoint http://morph-chain.frostfs.devenv:30333/ --resolve_order nns,dns
|
||||
```
|
||||
|
||||
### Processing of requests
|
||||
|
|
@ -80,18 +80,18 @@ Maximum number of clients whose requests can be handled by the gateway can be sp
|
|||
`--max_clients_deadline` defines deadline after which the gate sends error `RequestTimeout` to a client.
|
||||
|
||||
```shell
|
||||
$ neofs-s3-gw --max_clients_count 150 --max_clients_deadline 1m
|
||||
$ frostfs-s3-gw --max_clients_count 150 --max_clients_deadline 1m
|
||||
```
|
||||
|
||||
### Connection to NeoFS
|
||||
### Connection to FrostFS
|
||||
|
||||
Timeout to connect to NeoFS nodes can be set with `--connect_timeout`
|
||||
Timeout to connect to FrostFS nodes can be set with `--connect_timeout`
|
||||
and timeout to check node health during rebalance`--healthcheck_timeout`.
|
||||
|
||||
Also, interval to check node health can be specified by `--rebalance_interval` value.
|
||||
|
||||
```shell
|
||||
$ neofs-s3-gw --healthcheck_timeout 15s --connect_timeout 1m --rebalance_interval 1h
|
||||
$ frostfs-s3-gw --healthcheck_timeout 15s --connect_timeout 1m --rebalance_interval 1h
|
||||
```
|
||||
|
||||
### Monitoring and metrics
|
||||
|
|
@ -107,7 +107,7 @@ Examples of environment variables: [env-example](/config/config.env).
|
|||
A path to a configuration file can be specified with `--config` parameter:
|
||||
|
||||
```shell
|
||||
$ neofs-s3-gw --config your-config.yaml
|
||||
$ frostfs-s3-gw --config your-config.yaml
|
||||
```
|
||||
|
||||
### Reload on SIGHUP
|
||||
|
|
@ -124,12 +124,12 @@ $ kill -s SIGHUP <app_pid>
|
|||
Example:
|
||||
|
||||
```shell
|
||||
$ ./bin/neofs-s3-gw --config config.yaml &> s3.log &
|
||||
$ ./bin/frostfs-s3-gw --config config.yaml &> s3.log &
|
||||
[1] 998346
|
||||
|
||||
$ cat s3.log
|
||||
# ...
|
||||
2022-09-30T17:38:22.338+0300 info s3-gw/app.go:371 application started {"name": "neofs-s3-gw", "version": "v0.24.0"}
|
||||
2022-09-30T17:38:22.338+0300 info s3-gw/app.go:371 application started {"name": "frostfs-s3-gw", "version": "v0.24.0"}
|
||||
# ...
|
||||
|
||||
$ kill -s SIGHUP 998346
|
||||
|
|
@ -139,9 +139,9 @@ $ cat s3.log
|
|||
2022-09-30T17:38:40.909+0300 info s3-gw/app.go:491 SIGHUP config reload completed
|
||||
```
|
||||
|
||||
### NeoFS S3 Gateway configuration file
|
||||
### FrostFS S3 Gateway configuration file
|
||||
|
||||
This section contains detailed NeoFS S3 Gateway configuration file description
|
||||
This section contains detailed FrostFS S3 Gateway configuration file description
|
||||
including default config values and some tips to set up configurable values.
|
||||
|
||||
There are some custom types used for brevity:
|
||||
|
|
@ -165,16 +165,16 @@ There are some custom types used for brevity:
|
|||
| `cors` | [CORS configuration](#cors-section) |
|
||||
| `pprof` | [Pprof configuration](#pprof-section) |
|
||||
| `prometheus` | [Prometheus configuration](#prometheus-section) |
|
||||
| `neofs` | [Parameters of requests to NeoFS](#neofs-section) |
|
||||
| `neofs` | [Parameters of requests to FrostFS](#neofs-section) |
|
||||
|
||||
### General section
|
||||
|
||||
```yaml
|
||||
listen_domains:
|
||||
- s3dev.neofs.devenv
|
||||
- s3dev2.neofs.devenv
|
||||
- s3dev.frostfs.devenv
|
||||
- s3dev2.frostfs.devenv
|
||||
|
||||
rpc_endpoint: http://morph-chain.neofs.devenv:30333
|
||||
rpc_endpoint: http://morph-chain.frostfs.devenv:30333
|
||||
resolve_order:
|
||||
- nns
|
||||
- dns
|
||||
|
|
@ -226,23 +226,23 @@ wallet:
|
|||
|
||||
```yaml
|
||||
# Nodes configuration
|
||||
# This configuration makes the gateway use the first node (node1.neofs:8080)
|
||||
# while it's healthy. Otherwise, gateway uses the second node (node2.neofs:8080)
|
||||
# for 10% of requests and the third node (node3.neofs:8080) for 90% of requests.
|
||||
# This configuration makes the gateway use the first node (node1.frostfs:8080)
|
||||
# while it's healthy. Otherwise, gateway uses the second node (node2.frostfs:8080)
|
||||
# for 10% of requests and the third node (node3.frostfs:8080) for 90% of requests.
|
||||
# Until nodes with the same priority level are healthy
|
||||
# nodes with other priority are not used.
|
||||
# The lower the value, the higher the priority.
|
||||
peers:
|
||||
0:
|
||||
address: node1.neofs:8080
|
||||
address: node1.frostfs:8080
|
||||
priority: 1
|
||||
weight: 1
|
||||
1:
|
||||
address: node2.neofs:8080
|
||||
address: node2.frostfs:8080
|
||||
priority: 2
|
||||
weight: 0.1
|
||||
2:
|
||||
address: node3.neofs:8080
|
||||
address: node3.frostfs:8080
|
||||
priority: 2
|
||||
weight: 0.9
|
||||
```
|
||||
|
|
@ -263,9 +263,9 @@ placement_policy:
|
|||
```
|
||||
|
||||
| Parameter | Type | SIGHUP reload | Default value | Description |
|
||||
|------------------|----------|---------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `default` | `string` | yes | `REP 3` | Default policy of placing containers in NeoFS. If a user sends a request `CreateBucket` and doesn't define policy for placing of a container in NeoFS, the S3 Gateway will put the container with default policy. |
|
||||
| `region_mapping` | `string` | yes | | Path to file that maps aws `LocationContraint` values to NeoFS placement policy. The similar to `--container-policy` flag in `neofs-s3-authmate` util. |
|
||||
|------------------|----------|---------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `default` | `string` | yes | `REP 3` | Default policy of placing containers in FrostFS. If a user sends a request `CreateBucket` and doesn't define policy for placing of a container in FrostFS, the S3 Gateway will put the container with default policy. |
|
||||
| `region_mapping` | `string` | yes | | Path to file that maps aws `LocationContraint` values to FrostFS placement policy. The similar to `--container-policy` flag in `frostfs-s3-authmate` util. |
|
||||
|
||||
File for `region_mapping` must contain something like this:
|
||||
|
||||
|
|
@ -320,7 +320,7 @@ logger:
|
|||
|
||||
```yaml
|
||||
tree:
|
||||
service: s01.neofs.devenv:8080
|
||||
service: s01.frostfs.devenv:8080
|
||||
```
|
||||
|
||||
| Parameter | Type | Default value | Description |
|
||||
|
|
@ -356,7 +356,7 @@ cache:
|
|||
|
||||
| Parameter | Type | Default value | Description |
|
||||
|-----------------|-----------------------------------|-----------------------------------|----------------------------------------------------------------------------------------|
|
||||
| `objects` | [Cache config](#cache-subsection) | `lifetime: 5m`<br>`size: 1000000` | Cache for objects (NeoFS headers). |
|
||||
| `objects` | [Cache config](#cache-subsection) | `lifetime: 5m`<br>`size: 1000000` | Cache for objects (FrostFS headers). |
|
||||
| `list` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 100000` | Cache which keeps lists of objects in buckets. |
|
||||
| `names` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 10000` | Cache which contains mapping of nice name to object addresses. |
|
||||
| `buckets` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 1000` | Cache which contains mapping of bucket name to bucket info. |
|
||||
|
|
@ -449,14 +449,14 @@ prometheus:
|
|||
|
||||
# `neofs` section
|
||||
|
||||
Contains parameters of requests to NeoFS.
|
||||
Contains parameters of requests to FrostFS.
|
||||
This value can be overridden with `X-Amz-Meta-Neofs-Copies-Number` header for `PutObject`, `CopyObject`, `CreateMultipartUpload`.
|
||||
|
||||
```yaml
|
||||
neofs:
|
||||
frostfs:
|
||||
set_copies_number: 0
|
||||
```
|
||||
|
||||
| Parameter | Type | Default value | Description |
|
||||
|---------------------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `set_copies_number` | `uint32` | `0` | Number of the object copies to consider PUT to NeoFS successful. <br/>Default value `0` means that object will be processed according to the container's placement policy |
|
||||
|---------------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `set_copies_number` | `uint32` | `0` | Number of the object copies to consider PUT to FrostFS successful. <br/>Default value `0` means that object will be processed according to the container's placement policy |
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# S3 compatibility test results
|
||||
|
||||
NeoFS Node: v0.30.0
|
||||
NeoFS S3 Gateway: v0.22.0-10-g87f6681
|
||||
FrostFS Node: v0.30.0
|
||||
FrostFS S3 Gateway: v0.22.0-10-g87f6681
|
||||
|
||||
To update this file using tests result, run:
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# Tree service
|
||||
|
||||
To get objects' metadata and system information, the S3 GW makes requests to the Tree service.
|
||||
This is a service in NeoFS storage that keeps different information as a tree structure.
|
||||
This is a service in FrostFS storage that keeps different information as a tree structure.
|
||||
|
||||
Each node keeps one of the types of data as a set of **key-value pairs**:
|
||||
* Bucket settings: lock configuration and versioning mode
|
||||
|
|
@ -11,7 +11,7 @@ Each node keeps one of the types of data as a set of **key-value pairs**:
|
|||
* Object locking settings
|
||||
* Active multipart upload info
|
||||
|
||||
Some data takes up a lot of memory, so we store it in NeoFS nodes as an object with payload.
|
||||
Some data takes up a lot of memory, so we store it in FrostFS nodes as an object with payload.
|
||||
But we keep these objects' metadata in the Tree service too:
|
||||
* Notification configuration
|
||||
* CORS
|
||||
|
|
|
|||
Loading…
Reference in a new issue