TrueCloudLab
/
frostfs-s3-gw
16
2
Fork 14
Code Issues 19 Pull Requests 4 Actions Packages Releases 6 Activity

[#2] Update docs 

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
pull/4/head
Denis Kirillov 2022-12-15 16:37:18 +03:00 committed by Alex Vanin
parent 96dff367db
commit 09c6e22b84
8 changed files with 89 additions and 89 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/CODEOWNERS vendored
View File

@ -1 +1 @@
* @alexvanin @masterSplinter01 @KirillovDenis * @alexvanin @KirillovDenis

18
CONTRIBUTING.md
View File

@ -3,8 +3,8 @@
First, thank you for contributing! We love and encourage pull requests from First, thank you for contributing! We love and encourage pull requests from
everyone. Please follow the guidelines: everyone. Please follow the guidelines:
- Check the open [issues](https://github.com/nspcc-dev/neofs-s3-gw/issues) and - Check the open [issues](https://github.com/TrueCloudLab/frostfs-s3-gw/issues) and
[pull requests](https://github.com/nspcc-dev/neofs-s3-gw/pulls) for existing [pull requests](https://github.com/TrueCloudLab/frostfs-s3-gw/pulls) for existing
discussions. discussions.
- Open an issue first, to discuss a new feature or enhancement. - Open an issue first, to discuss a new feature or enhancement.
@ -23,24 +23,24 @@ everyone. Please follow the guidelines:
## Development Workflow ## Development Workflow
Start by forking the `neofs-s3-gw` repository, make changes in a branch and then Start by forking the `frostfs-s3-gw` repository, make changes in a branch and then
send a pull request. We encourage pull requests to discuss code changes. Here send a pull request. We encourage pull requests to discuss code changes. Here
are the steps in details: are the steps in details:
### Set up your GitHub Repository ### Set up your GitHub Repository
Fork [NeoFS S3 Gateway Fork [FrostFS S3 Gateway
upstream](https://github.com/nspcc-dev/neofs-s3-gw/fork) source repository upstream](https://github.com/TrueCloudLab/frostfs-s3-gw/fork) source repository
to your own personal repository. Copy the URL of your fork (you will need it for to your own personal repository. Copy the URL of your fork (you will need it for
the `git clone` command below). the `git clone` command below).
```sh ```sh
$ git clone https://github.com/nspcc-dev/neofs-s3-gw $ git clone https://github.com/TrueCloudLab/frostfs-s3-gw
``` ```
### Set up git remote as ``upstream`` ### Set up git remote as ``upstream``
```sh ```sh
$ cd neofs-s3-gw $ cd frostfs-s3-gw
$ git remote add upstream https://github.com/nspcc-dev/neofs-s3-gw $ git remote add upstream https://github.com/TrueCloudLab/frostfs-s3-gw
$ git fetch upstream $ git fetch upstream
$ git merge upstream/master $ git merge upstream/master
... ...
@ -107,7 +107,7 @@ contributors".
To sign your work, just add a line like this at the end of your commit message: To sign your work, just add a line like this at the end of your commit message:
``` ```
Signed-off-by: Samii Sakisaka <samii@nspcc.ru> Signed-off-by: Samii Sakisaka <samii@frostfs.info>
``` ```
This can be easily done with the `--signoff` option to `git commit`. This can be easily done with the `--signoff` option to `git commit`.

30
README.md
View File

@ -1,13 +1,13 @@
# NeoFS S3 Gateway # FrostFS S3 Gateway
NeoFS S3 gateway provides API compatible with Amazon S3 cloud storage service. FrostFS S3 gateway provides API compatible with Amazon S3 cloud storage service.
## Installation ## Installation
```go get -u github.com/nspcc-dev/neofs-s3-gw``` ```go get -u github.com/TrueCloudLab/frostfs-s3-gw```
Or you can call `make` to build it from the cloned repository (the binary will Or you can call `make` to build it from the cloned repository (the binary will
end up in `bin/neofs-s3-gw` with authmate helper in `bin/neofs-s3-authmate`). end up in `bin/frostfs-s3-gw` with authmate helper in `bin/frostfs-s3-authmate`).
To build binaries in clean docker environment, call `make docker/all`. To build binaries in clean docker environment, call `make docker/all`.
Other notable make targets: Other notable make targets:
@ -22,36 +22,36 @@ version Show current version
``` ```
Or you can also use a [Docker Or you can also use a [Docker
image](https://hub.docker.com/r/nspccdev/neofs-s3-gw) provided for released image](https://hub.docker.com/r/nspccdev/frostfs-s3-gw) provided for released
(and occasionally unreleased) versions of gateway (`:latest` points to the (and occasionally unreleased) versions of gateway (`:latest` points to the
latest stable release). latest stable release).
## Execution ## Execution
Minimalistic S3 gateway setup needs: Minimalistic S3 gateway setup needs:
* NeoFS node(s) address (S3 gateway itself is not a NeoFS node) * FrostFS node(s) address (S3 gateway itself is not a FrostFS node)
Passed via `-p` parameter or via `S3_GW_PEERS_<N>_ADDRESS` and Passed via `-p` parameter or via `S3_GW_PEERS_<N>_ADDRESS` and
`S3_GW_PEERS_<N>_WEIGHT` environment variables (gateway supports multiple `S3_GW_PEERS_<N>_WEIGHT` environment variables (gateway supports multiple
NeoFS nodes with weighted load balancing). FrostFS nodes with weighted load balancing).
* a wallet used to fetch key and communicate with NeoFS nodes * a wallet used to fetch key and communicate with FrostFS nodes
Passed via `--wallet` parameter or `S3_GW_WALLET_PATH` environment variable. Passed via `--wallet` parameter or `S3_GW_WALLET_PATH` environment variable.
These two commands are functionally equivalent, they run the gate with one These two commands are functionally equivalent, they run the gate with one
backend node, some keys and otherwise default settings: backend node, some keys and otherwise default settings:
``` ```
$ neofs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json $ frostfs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json
$ S3_GW_PEERS_0_ADDRESS=192.168.130.72:8080 \ $ S3_GW_PEERS_0_ADDRESS=192.168.130.72:8080 \
S3_GW_WALLET=wallet.json \ S3_GW_WALLET=wallet.json \
neofs-s3-gw frostfs-s3-gw
``` ```
It's also possible to specify uri scheme (grpc or grpcs) when using `-p` or environment variables: It's also possible to specify uri scheme (grpc or grpcs) when using `-p` or environment variables:
``` ```
$ neofs-s3-gw -p grpc://192.168.130.72:8080 --wallet wallet.json $ frostfs-s3-gw -p grpc://192.168.130.72:8080 --wallet wallet.json
$ S3_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 \ $ S3_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 \
S3_GW_WALLET=wallet.json \ S3_GW_WALLET=wallet.json \
neofs-s3-gw frostfs-s3-gw
``` ```
## Domains ## Domains
@ -60,7 +60,7 @@ By default, s3-gw enable only `path-style access`.
To be able to use both: `virtual-hosted-style` and `path-style` access you must configure `listen_domains`: To be able to use both: `virtual-hosted-style` and `path-style` access you must configure `listen_domains`:
```shell ```shell
$ neofs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json --listen_domains your.first.domain --listen_domains your.second.domain $ frostfs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json --listen_domains your.first.domain --listen_domains your.second.domain
``` ```
So now you can use (e.g. `HeadBucket`. Make sure DNS is properly configured): So now you can use (e.g. `HeadBucket`. Make sure DNS is properly configured):
@ -84,8 +84,8 @@ Also, you can configure domains using `.env` variables or `yaml` file.
## Documentation ## Documentation
- [Configuration](./docs/configuration.md) - [Configuration](./docs/configuration.md)
- [NeoFS S3 AuthMate](./docs/authmate.md) - [FrostFS S3 AuthMate](./docs/authmate.md)
- [NeoFS Tree service](./docs/tree_service.md) - [FrostFS Tree service](./docs/tree_service.md)
- [AWS CLI basic usage](./docs/aws_cli.md) - [AWS CLI basic usage](./docs/aws_cli.md)
- [AWS S3 API compatibility](./docs/aws_s3_compat.md) - [AWS S3 API compatibility](./docs/aws_s3_compat.md)
- [AWS S3 Compatibility test results](./docs/s3_test_results.md) - [AWS S3 Compatibility test results](./docs/s3_test_results.md)

36
docs/authmate.md
View File

@ -1,18 +1,18 @@
# NeoFS S3 AuthMate # FrostFS S3 AuthMate
Authmate is a tool to create gateway AWS credentials. AWS users Authmate is a tool to create gateway AWS credentials. AWS users
are authenticated with access key IDs and secrets, while NeoFS users are are authenticated with access key IDs and secrets, while FrostFS users are
authenticated with key pairs. To complicate things further, we have S3 gateway authenticated with key pairs. To complicate things further, we have S3 gateway
that usually acts on behalf of some user, but the user doesn't necessarily want to that usually acts on behalf of some user, but the user doesn't necessarily want to
give their keys to the gateway. give their keys to the gateway.
To solve this, we use NeoFS bearer tokens that are signed by the owner (NeoFS To solve this, we use FrostFS bearer tokens that are signed by the owner (FrostFS
"user") and that can implement any kind of policy for NeoFS requests allowed "user") and that can implement any kind of policy for FrostFS requests allowed
to use this token. However, tokens can't be used as AWS credentials directly. Thus, to use this token. However, tokens can't be used as AWS credentials directly. Thus,
they're stored on NeoFS as regular objects, and an access key ID is just an they're stored on FrostFS as regular objects, and an access key ID is just an
address of this object while a secret is generated randomly. address of this object while a secret is generated randomly.
Tokens are not stored on NeoFS in plaintext, they're encrypted with a set of Tokens are not stored on FrostFS in plaintext, they're encrypted with a set of
gateway keys. So, in order for a gateway to be able to successfully extract bearer gateway keys. So, in order for a gateway to be able to successfully extract bearer
token, the object needs to be stored in a container available for the gateway token, the object needs to be stored in a container available for the gateway
to read, and it needs to be encrypted with this gateway's key (among others to read, and it needs to be encrypted with this gateway's key (among others
@ -83,7 +83,7 @@ NhLQpDnerpviUWDF77j5qyjFgavCmasJ4p (simple signature contract):
## Issuance of a secret ## Issuance of a secret
To issue a secret means to create Bearer and, optionally, Session tokens and To issue a secret means to create Bearer and, optionally, Session tokens and
put them as an object into a container on the NeoFS network. put them as an object into a container on the FrostFS network.
### CLI parameters ### CLI parameters
@ -91,7 +91,7 @@ put them as an object into a container on the NeoFS network.
* `--wallet` is a path to a wallet `.json` file. You can provide a passphrase to decrypt * `--wallet` is a path to a wallet `.json` file. You can provide a passphrase to decrypt
a wallet via environment variable `AUTHMATE_WALLET_PASSPHRASE`, or you will be asked to enter a passphrase a wallet via environment variable `AUTHMATE_WALLET_PASSPHRASE`, or you will be asked to enter a passphrase
interactively. You can also specify an account address to use from a wallet using the `--address` parameter. interactively. You can also specify an account address to use from a wallet using the `--address` parameter.
* `--peer` is an address of a NeoFS peer to connect to * `--peer` is an address of a FrostFS peer to connect to
* `--gate-public-key` is a public `secp256r1` 33-byte short key of a gate (use flags repeatedly for multiple gates). The tokens are encrypted * `--gate-public-key` is a public `secp256r1` 33-byte short key of a gate (use flags repeatedly for multiple gates). The tokens are encrypted
by a set of gateway keys, so you need to pass them as well. by a set of gateway keys, so you need to pass them as well.
@ -105,7 +105,7 @@ You can issue a secret using the parameters above only. The tool will
E.g.: E.g.:
```shell ```shell
$ neofs-s3-authmate issue-secret --wallet wallet.json \ $ frostfs-s3-authmate issue-secret --wallet wallet.json \
--peer 192.168.130.71:8080 \ --peer 192.168.130.71:8080 \
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf\ --gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf\
--gate-public-key 0317585fa8274f7afdf1fc5f2a2e7bece549d5175c4e5182e37924f30229aef967 --gate-public-key 0317585fa8274f7afdf1fc5f2a2e7bece549d5175c4e5182e37924f30229aef967
@ -122,7 +122,7 @@ $ neofs-s3-authmate issue-secret --wallet wallet.json \
`access_key_id` and `secret_access_key` are AWS credentials that you can use with any S3 client. `access_key_id` and `secret_access_key` are AWS credentials that you can use with any S3 client.
`access_key_id` consists of Base58 encoded containerID(cid) and objectID(oid) stored on the NeoFS network and containing `access_key_id` consists of Base58 encoded containerID(cid) and objectID(oid) stored on the FrostFS network and containing
the secret. Format of `access_key_id`: `%cid0%oid`, where 0(zero) is a delimiter. the secret. Format of `access_key_id`: `%cid0%oid`, where 0(zero) is a delimiter.
**Optional parameters:** **Optional parameters:**
@ -141,7 +141,7 @@ Creation of bearer tokens is mandatory.
Rules for a bearer token can be set via parameter `--bearer-rules` (json-string and file path allowed): Rules for a bearer token can be set via parameter `--bearer-rules` (json-string and file path allowed):
```shell ```shell
$ neofs-s3-authmate issue-secret --wallet wallet.json \ $ frostfs-s3-authmate issue-secret --wallet wallet.json \
--peer 192.168.130.71:8080 \ --peer 192.168.130.71:8080 \
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \ --gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \
--bearer-rules bearer-rules.json --bearer-rules bearer-rules.json
@ -195,7 +195,7 @@ If bearer rules are not set, a token will be auto-generated with a value:
With a session token, there are 3 options: With a session token, there are 3 options:
1. append `--session-tokens` parameter with your custom rules in json format (as a string or file path). E.g.: 1. append `--session-tokens` parameter with your custom rules in json format (as a string or file path). E.g.:
```shell ```shell
$ neofs-s3-authmate issue-secret --wallet wallet.json \ $ frostfs-s3-authmate issue-secret --wallet wallet.json \
--peer 192.168.130.71:8080 \ --peer 192.168.130.71:8080 \
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \ --gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \
--session-tokens session.json --session-tokens session.json
@ -224,7 +224,7 @@ If `containerID` is `null` or omitted, then session token rule will be applied
to all containers. Otherwise, specify `containerID` value in human-redabale to all containers. Otherwise, specify `containerID` value in human-redabale
format (base58 encoded string). format (base58 encoded string).
> **_NB!_** To create buckets in NeoFS it's necessary to have session tokens with `PUT` and `SETEACL` permissions, that's why > **_NB!_** To create buckets in FrostFS it's necessary to have session tokens with `PUT` and `SETEACL` permissions, that's why
the authmate creates a `SETEACL` session token automatically in case when a user specified the token rule with `PUT` and the authmate creates a `SETEACL` session token automatically in case when a user specified the token rule with `PUT` and
forgot about the rule with `SETEACL`. forgot about the rule with `SETEACL`.
@ -235,7 +235,7 @@ in example above)
### Containers policy ### Containers policy
Rules for mapping of `LocationConstraint` ([aws spec](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html#API_CreateBucket_RequestBody)) Rules for mapping of `LocationConstraint` ([aws spec](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html#API_CreateBucket_RequestBody))
to `PlacementPolicy` ([neofs spec](https://github.com/nspcc-dev/neofs-spec/blob/master/01-arch/02-policy.md)) to `PlacementPolicy` ([frostfs spec](https://github.com/TrueCloudLab/frostfs-spec/blob/master/01-arch/02-policy.md))
can be set via parameter `--container-policy` (json-string and file path allowed): can be set via parameter `--container-policy` (json-string and file path allowed):
```json ```json
{ {
@ -248,12 +248,12 @@ can be set via parameter `--container-policy` (json-string and file path allowed
## Obtainment of a secret access key ## Obtainment of a secret access key
You can get a secret access key associated with an access key ID by obtaining a You can get a secret access key associated with an access key ID by obtaining a
secret stored on the NeoFS network. Here is an example of providing one password (for `wallet.json`) via env variable secret stored on the FrostFS network. Here is an example of providing one password (for `wallet.json`) via env variable
and the other (for `gate-wallet.json`) interactively: and the other (for `gate-wallet.json`) interactively:
```shell ```shell
$ AUTHMATE_WALLET_PASSPHRASE=some-pwd \ $ AUTHMATE_WALLET_PASSPHRASE=some-pwd \
neofs-s3-authmate obtain-secret --wallet wallet.json \ frostfs-s3-authmate obtain-secret --wallet wallet.json \
--peer 192.168.130.71:8080 \ --peer 192.168.130.71:8080 \
--gate-wallet gate-wallet.json \ --gate-wallet gate-wallet.json \
--access-key-id 5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM --access-key-id 5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM
@ -272,7 +272,7 @@ using AWS credentials from `~/.aws/credentials` (you can specify profile using t
with the following command: with the following command:
```shell ```shell
$ neofs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \ $ frostfs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \
--method get --bucket presigned --object obj --lifetime 30s --method get --bucket presigned --object obj --lifetime 30s
{ {
@ -283,7 +283,7 @@ $ neofs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \
You can also provide credential explicitly: You can also provide credential explicitly:
```shell ```shell
$ neofs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \ $ frostfs-s3-authmate generate-presigned-url --endpoint http://localhost:8084 \
--method put --bucket presigned --object obj --lifetime 12h \ --method put --bucket presigned --object obj --lifetime 12h \
--region ru --aws-secret-access-key c2d65ef2980f03f4f495bdebedeeae760496697880d61d106bb9a4e5cd2e0607 \ --region ru --aws-secret-access-key c2d65ef2980f03f4f495bdebedeeae760496697880d61d106bb9a4e5cd2e0607 \
--aws-access-key-id ETaA2CadPcA7bAkLsML2PbTudXY8uRt2PDjCCwkvRv9s0FDCxWDXYc1SA1vKv8KbyCNsLY2AmAjJ92Vz5rgvsFCy --aws-access-key-id ETaA2CadPcA7bAkLsML2PbTudXY8uRt2PDjCCwkvRv9s0FDCxWDXYc1SA1vKv8KbyCNsLY2AmAjJ92Vz5rgvsFCy

12
docs/aws_cli.md
View File

@ -6,7 +6,7 @@
To configure basic settings that the AWS CLI uses to interact with the Gateway, follow the steps below: To configure basic settings that the AWS CLI uses to interact with the Gateway, follow the steps below:
1. issue a secret with neofs-s3-authmate tool (see [NeoFS S3 Authmate](./authmate.md)) 1. issue a secret with frostfs-s3-authmate tool (see [FrostFS S3 Authmate](./authmate.md))
2. execute the command 2. execute the command
``` ```
$ aws configure $ aws configure
@ -28,7 +28,7 @@ Default output format [none]: json
#### Obtainment of a list of buckets #### Obtainment of a list of buckets
To view the list of the buckets in the NeoFS node, to which the gateway is connected, enter the following command: To view the list of the buckets in the FrostFS node, to which the gateway is connected, enter the following command:
``` ```
$ aws s3 ls $ aws s3 ls
``` ```
@ -72,15 +72,15 @@ $ aws s3api list-objects --bucket %BUCKET_NAME
#### Upload of a file #### Upload of a file
To upload a file into a bucket in the NeoFS network, run the following command: To upload a file into a bucket in the FrostFS network, run the following command:
``` ```
$ aws s3api put-object --bucket %BUCKET_NAME --key %OBJECT_KEY --body %FILEPATH $ aws s3api put-object --bucket %BUCKET_NAME --key %OBJECT_KEY --body %FILEPATH
``` ```
where %OBJECT_KEY is the filepath of an object in NeoFS where %OBJECT_KEY is the filepath of an object in FrostFS
#### Upload of a dir #### Upload of a dir
To upload a dir into a bucket in the NeoFS network, run the following command: To upload a dir into a bucket in the FrostFS network, run the following command:
``` ```
$ aws s3 sync %DIRPATH s3://%BUCKET_NAME $ aws s3 sync %DIRPATH s3://%BUCKET_NAME
@ -88,7 +88,7 @@ $ aws s3 sync %DIRPATH s3://%BUCKET_NAME
#### Download of a file #### Download of a file
To download a file from a bucket in the NeoFS Network, execute: To download a file from a bucket in the FrostFS Network, execute:
``` ```
$ aws s3api get-object --bucket %BUCKET_NAME --key %OBJECT_KEY %OUTFILE $ aws s3api get-object --bucket %BUCKET_NAME --key %OBJECT_KEY %OUTFILE
``` ```

72
docs/configuration.md
View File

@ -19,20 +19,20 @@ basic configuration can be completed with CLI parameters only.
3. [Binding and TLS](#listening-on-address-and-TLS) 3. [Binding and TLS](#listening-on-address-and-TLS)
4. [RPC endpoint and resolving of bucket names](#rpc-endpoint-and-resolving-of-bucket-names) 4. [RPC endpoint and resolving of bucket names](#rpc-endpoint-and-resolving-of-bucket-names)
5. [Processing of requests](#processing-of-requests) 5. [Processing of requests](#processing-of-requests)
6. [Connection to NeoFS](#connection-to-NeoFS) 6. [Connection to FrostFS](#connection-to-FrostFS)
7. [Monitoring and metrics](#monitoring-and-metrics) 7. [Monitoring and metrics](#monitoring-and-metrics)
2. [YAML file and environment variables](#yaml-file-and-environment-variables) 2. [YAML file and environment variables](#yaml-file-and-environment-variables)
1. [Configuration file](#neofs-s3-gateway-configuration-file) 1. [Configuration file](#frostfs-s3-gateway-configuration-file)
## CLI parameters ## CLI parameters
### Nodes and weights ### Nodes and weights
You can specify multiple `-p` options to add more NeoFS nodes; this will make You can specify multiple `-p` options to add more FrostFS nodes; this will make
a gateway spread requests equally among them (using weight 1 for every node): a gateway spread requests equally among them (using weight 1 for every node):
```shell ```shell
$ neofs-s3-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080 $ frostfs-s3-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080
``` ```
If you want some specific load distribution proportions, use weights and priorities, they If you want some specific load distribution proportions, use weights and priorities, they
@ -58,7 +58,7 @@ Example to bind to `192.168.130.130:443` and serve TLS there (keys and nodes are
omitted): omitted):
```shell ```shell
$ neofs-s3-gw --listen_address 192.168.130.130:443 \ $ frostfs-s3-gw --listen_address 192.168.130.130:443 \
--tls.key_file=key.pem --tls.cert_file=cert.pem --tls.key_file=key.pem --tls.cert_file=cert.pem
``` ```
@ -70,7 +70,7 @@ To set RPC endpoint specify a value of parameter `-r` or `--rpc_endpoint`. The p
parameter's `--resolve_order` value contains `nns`. parameter's `--resolve_order` value contains `nns`.
```shell ```shell
$ neofs-s3-gw --rpc_endpoint http://morph-chain.neofs.devenv:30333/ --resolve_order nns,dns $ frostfs-s3-gw --rpc_endpoint http://morph-chain.frostfs.devenv:30333/ --resolve_order nns,dns
``` ```
### Processing of requests ### Processing of requests
@ -80,18 +80,18 @@ Maximum number of clients whose requests can be handled by the gateway can be sp
`--max_clients_deadline` defines deadline after which the gate sends error `RequestTimeout` to a client. `--max_clients_deadline` defines deadline after which the gate sends error `RequestTimeout` to a client.
```shell ```shell
$ neofs-s3-gw --max_clients_count 150 --max_clients_deadline 1m $ frostfs-s3-gw --max_clients_count 150 --max_clients_deadline 1m
``` ```
### Connection to NeoFS ### Connection to FrostFS
Timeout to connect to NeoFS nodes can be set with `--connect_timeout` Timeout to connect to FrostFS nodes can be set with `--connect_timeout`
and timeout to check node health during rebalance`--healthcheck_timeout`. and timeout to check node health during rebalance`--healthcheck_timeout`.
Also, interval to check node health can be specified by `--rebalance_interval` value. Also, interval to check node health can be specified by `--rebalance_interval` value.
```shell ```shell
$ neofs-s3-gw --healthcheck_timeout 15s --connect_timeout 1m --rebalance_interval 1h $ frostfs-s3-gw --healthcheck_timeout 15s --connect_timeout 1m --rebalance_interval 1h
``` ```
### Monitoring and metrics ### Monitoring and metrics
@ -107,7 +107,7 @@ Examples of environment variables: [env-example](/config/config.env).
A path to a configuration file can be specified with `--config` parameter: A path to a configuration file can be specified with `--config` parameter:
```shell ```shell
$ neofs-s3-gw --config your-config.yaml $ frostfs-s3-gw --config your-config.yaml
``` ```
### Reload on SIGHUP ### Reload on SIGHUP
@ -124,12 +124,12 @@ $ kill -s SIGHUP <app_pid>
Example: Example:
```shell ```shell
$ ./bin/neofs-s3-gw --config config.yaml &> s3.log & $ ./bin/frostfs-s3-gw --config config.yaml &> s3.log &
[1] 998346 [1] 998346
$ cat s3.log $ cat s3.log
# ... # ...
2022-09-30T17:38:22.338+0300 info s3-gw/app.go:371 application started {"name": "neofs-s3-gw", "version": "v0.24.0"} 2022-09-30T17:38:22.338+0300 info s3-gw/app.go:371 application started {"name": "frostfs-s3-gw", "version": "v0.24.0"}
# ... # ...
$ kill -s SIGHUP 998346 $ kill -s SIGHUP 998346
@ -139,9 +139,9 @@ $ cat s3.log
2022-09-30T17:38:40.909+0300 info s3-gw/app.go:491 SIGHUP config reload completed 2022-09-30T17:38:40.909+0300 info s3-gw/app.go:491 SIGHUP config reload completed
``` ```
### NeoFS S3 Gateway configuration file ### FrostFS S3 Gateway configuration file
This section contains detailed NeoFS S3 Gateway configuration file description This section contains detailed FrostFS S3 Gateway configuration file description
including default config values and some tips to set up configurable values. including default config values and some tips to set up configurable values.
There are some custom types used for brevity: There are some custom types used for brevity:
@ -165,16 +165,16 @@ There are some custom types used for brevity:
| `cors` | [CORS configuration](#cors-section) | | `cors` | [CORS configuration](#cors-section) |
| `pprof` | [Pprof configuration](#pprof-section) | | `pprof` | [Pprof configuration](#pprof-section) |
| `prometheus` | [Prometheus configuration](#prometheus-section) | | `prometheus` | [Prometheus configuration](#prometheus-section) |
| `neofs` | [Parameters of requests to NeoFS](#neofs-section) | | `neofs` | [Parameters of requests to FrostFS](#neofs-section) |
### General section ### General section
```yaml ```yaml
listen_domains: listen_domains:
- s3dev.neofs.devenv - s3dev.frostfs.devenv
- s3dev2.neofs.devenv - s3dev2.frostfs.devenv
rpc_endpoint: http://morph-chain.neofs.devenv:30333 rpc_endpoint: http://morph-chain.frostfs.devenv:30333
resolve_order: resolve_order:
- nns - nns
- dns - dns
@ -226,23 +226,23 @@ wallet:
```yaml ```yaml
# Nodes configuration # Nodes configuration
# This configuration makes the gateway use the first node (node1.neofs:8080) # This configuration makes the gateway use the first node (node1.frostfs:8080)
# while it's healthy. Otherwise, gateway uses the second node (node2.neofs:8080) # while it's healthy. Otherwise, gateway uses the second node (node2.frostfs:8080)
# for 10% of requests and the third node (node3.neofs:8080) for 90% of requests. # for 10% of requests and the third node (node3.frostfs:8080) for 90% of requests.
# Until nodes with the same priority level are healthy # Until nodes with the same priority level are healthy
# nodes with other priority are not used. # nodes with other priority are not used.
# The lower the value, the higher the priority. # The lower the value, the higher the priority.
peers: peers:
0: 0:
address: node1.neofs:8080 address: node1.frostfs:8080
priority: 1 priority: 1
weight: 1 weight: 1
1: 1:
address: node2.neofs:8080 address: node2.frostfs:8080
priority: 2 priority: 2
weight: 0.1 weight: 0.1
2: 2:
address: node3.neofs:8080 address: node3.frostfs:8080
priority: 2 priority: 2
weight: 0.9 weight: 0.9
``` ```
@ -262,10 +262,10 @@ placement_policy:
region_mapping: /path/to/mapping/rules.json region_mapping: /path/to/mapping/rules.json
``` ```
| Parameter | Type | SIGHUP reload | Default value | Description | | Parameter | Type | SIGHUP reload | Default value | Description |
|------------------|----------|---------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |------------------|----------|---------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `default` | `string` | yes | `REP 3` | Default policy of placing containers in NeoFS. If a user sends a request `CreateBucket` and doesn't define policy for placing of a container in NeoFS, the S3 Gateway will put the container with default policy. | | `default` | `string` | yes | `REP 3` | Default policy of placing containers in FrostFS. If a user sends a request `CreateBucket` and doesn't define policy for placing of a container in FrostFS, the S3 Gateway will put the container with default policy. |
| `region_mapping` | `string` | yes | | Path to file that maps aws `LocationContraint` values to NeoFS placement policy. The similar to `--container-policy` flag in `neofs-s3-authmate` util. | | `region_mapping` | `string` | yes | | Path to file that maps aws `LocationContraint` values to FrostFS placement policy. The similar to `--container-policy` flag in `frostfs-s3-authmate` util. |
File for `region_mapping` must contain something like this: File for `region_mapping` must contain something like this:
@ -320,7 +320,7 @@ logger:
```yaml ```yaml
tree: tree:
service: s01.neofs.devenv:8080 service: s01.frostfs.devenv:8080
``` ```
| Parameter | Type | Default value | Description | | Parameter | Type | Default value | Description |
@ -356,7 +356,7 @@ cache:
| Parameter | Type | Default value | Description | | Parameter | Type | Default value | Description |
|-----------------|-----------------------------------|-----------------------------------|----------------------------------------------------------------------------------------| |-----------------|-----------------------------------|-----------------------------------|----------------------------------------------------------------------------------------|
| `objects` | [Cache config](#cache-subsection) | `lifetime: 5m`<br>`size: 1000000` | Cache for objects (NeoFS headers). | | `objects` | [Cache config](#cache-subsection) | `lifetime: 5m`<br>`size: 1000000` | Cache for objects (FrostFS headers). |
| `list` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 100000` | Cache which keeps lists of objects in buckets. | | `list` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 100000` | Cache which keeps lists of objects in buckets. |
| `names` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 10000` | Cache which contains mapping of nice name to object addresses. | | `names` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 10000` | Cache which contains mapping of nice name to object addresses. |
| `buckets` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 1000` | Cache which contains mapping of bucket name to bucket info. | | `buckets` | [Cache config](#cache-subsection) | `lifetime: 60s`<br>`size: 1000` | Cache which contains mapping of bucket name to bucket info. |
@ -449,14 +449,14 @@ prometheus:
# `neofs` section # `neofs` section
Contains parameters of requests to NeoFS. Contains parameters of requests to FrostFS.
This value can be overridden with `X-Amz-Meta-Neofs-Copies-Number` header for `PutObject`, `CopyObject`, `CreateMultipartUpload`. This value can be overridden with `X-Amz-Meta-Neofs-Copies-Number` header for `PutObject`, `CopyObject`, `CreateMultipartUpload`.
```yaml ```yaml
neofs: frostfs:
set_copies_number: 0 set_copies_number: 0
``` ```
| Parameter | Type | Default value | Description | | Parameter | Type | Default value | Description |
|---------------------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |---------------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `set_copies_number` | `uint32` | `0` | Number of the object copies to consider PUT to NeoFS successful. <br/>Default value `0` means that object will be processed according to the container's placement policy | | `set_copies_number` | `uint32` | `0` | Number of the object copies to consider PUT to FrostFS successful. <br/>Default value `0` means that object will be processed according to the container's placement policy |

4
docs/s3_test_results.md
View File

@ -1,7 +1,7 @@
# S3 compatibility test results # S3 compatibility test results
NeoFS Node: v0.30.0 FrostFS Node: v0.30.0
NeoFS S3 Gateway: v0.22.0-10-g87f6681 FrostFS S3 Gateway: v0.22.0-10-g87f6681
To update this file using tests result, run: To update this file using tests result, run:

4
docs/tree_service.md
View File

@ -1,7 +1,7 @@
# Tree service # Tree service
To get objects' metadata and system information, the S3 GW makes requests to the Tree service. To get objects' metadata and system information, the S3 GW makes requests to the Tree service.
This is a service in NeoFS storage that keeps different information as a tree structure. This is a service in FrostFS storage that keeps different information as a tree structure.
Each node keeps one of the types of data as a set of **key-value pairs**: Each node keeps one of the types of data as a set of **key-value pairs**:
* Bucket settings: lock configuration and versioning mode * Bucket settings: lock configuration and versioning mode
@ -11,7 +11,7 @@ Each node keeps one of the types of data as a set of **key-value pairs**:
* Object locking settings * Object locking settings
* Active multipart upload info * Active multipart upload info
Some data takes up a lot of memory, so we store it in NeoFS nodes as an object with payload. Some data takes up a lot of memory, so we store it in FrostFS nodes as an object with payload.
But we keep these objects' metadata in the Tree service too: But we keep these objects' metadata in the Tree service too:
* Notification configuration * Notification configuration
* CORS * CORS