forked from TrueCloudLab/frostfs-s3-gw
[#210] *: Split README into docs
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
This commit is contained in:
parent
0791144284
commit
cfe92320c8
4 changed files with 388 additions and 388 deletions
392
README.md
392
README.md
|
@ -53,395 +53,11 @@ $ S3_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 \
|
||||||
neofs-s3-gw
|
neofs-s3-gw
|
||||||
```
|
```
|
||||||
|
|
||||||
## Configuration
|
## Documentation
|
||||||
|
|
||||||
In general, everything available as CLI parameter can also be specified via
|
- [Configuration](./docs/configuration.md)
|
||||||
environment variables, so they're not specifically mentioned in most cases
|
- [NeoFS AuthMate](./docs/authmate.md)
|
||||||
(see `--help` also). If you prefer a config file you can use it in yaml format.
|
- [AWS CLI basic usage](./docs/aws_cli.md)
|
||||||
|
|
||||||
### Nodes and weights
|
|
||||||
|
|
||||||
You can specify multiple `-p` options to add more NeoFS nodes, this will make
|
|
||||||
gateway spread requests equally among them (using weight 1 for every node):
|
|
||||||
|
|
||||||
```
|
|
||||||
$ neofs-s3-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080
|
|
||||||
```
|
|
||||||
If you want some specific load distribution proportions, use weights, but they
|
|
||||||
can only be specified via environment variables:
|
|
||||||
|
|
||||||
```
|
|
||||||
$ HTTP_GW_PEERS_0_ADDRESS=192.168.130.72:8080 HTTP_GW_PEERS_0_WEIGHT=9 \
|
|
||||||
HTTP_GW_PEERS_1_ADDRESS=192.168.130.71:8080 HTTP_GW_PEERS_1_WEIGHT=1 neofs-s3-gw
|
|
||||||
```
|
|
||||||
This command will make gateway use 192.168.130.72 for 90% of requests and
|
|
||||||
192.168.130.71 for remaining 10%.
|
|
||||||
|
|
||||||
### Key
|
|
||||||
|
|
||||||
Wallet (`--wallet`) is mandatory parameter. It is a path to wallet file. You can provide password to decrypt wallet
|
|
||||||
via `S3_GW_WALLET_PASSPHRASE` variable or you will be asked to enter the password interactively.
|
|
||||||
You also can specify account address to use from wallet using `--address` parameter.
|
|
||||||
|
|
||||||
### Binding and TLS
|
|
||||||
|
|
||||||
Gateway binds to `0.0.0.0:8080` by default and you can change that with
|
|
||||||
`--listen_address` option.
|
|
||||||
|
|
||||||
It can also provide TLS interface for its users, just specify paths to key and
|
|
||||||
certificate files via `--tls.key_file` and `--tls.cert_file` parameters. Note
|
|
||||||
that using these options makes gateway TLS-only, if you need to serve both TLS
|
|
||||||
and plain text you either have to run two gateway instances or use some
|
|
||||||
external redirecting solution.
|
|
||||||
|
|
||||||
Example to bind to `192.168.130.130:443` and serve TLS there (keys and nodes
|
|
||||||
omitted):
|
|
||||||
|
|
||||||
```
|
|
||||||
$ neofs-s3-gw --listen_address 192.168.130.130:443 \
|
|
||||||
--tls.key_file=key.pem --tls.cert_file=cert.pem
|
|
||||||
```
|
|
||||||
|
|
||||||
### Monitoring and metrics
|
|
||||||
|
|
||||||
Pprof and Prometheus are integrated into the gateway, but not enabled by
|
|
||||||
default. To enable them use `--pprof` and `--metrics` flags or
|
|
||||||
`HTTP_GW_PPROF`/`HTTP_GW_METRICS` environment variables.
|
|
||||||
|
|
||||||
### Yaml file
|
|
||||||
Configuration file is optional and can be used instead of environment variables/other parameters.
|
|
||||||
It can be specified with `--config` parameter:
|
|
||||||
```
|
|
||||||
$ neofs-s3-gw --config your-config.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
Configuration file example:
|
|
||||||
```
|
|
||||||
listen_address: 0.0.0.0:8084
|
|
||||||
|
|
||||||
wallet:
|
|
||||||
passphrase: 123456
|
|
||||||
|
|
||||||
logger:
|
|
||||||
level: debug
|
|
||||||
|
|
||||||
peers:
|
|
||||||
0:
|
|
||||||
address: s01.neofs.devenv:8080
|
|
||||||
weight: 1
|
|
||||||
```
|
|
||||||
|
|
||||||
To know nesting level of variable you need to cut off the prefix `S3_GW` from variable and split the rest parts by `_`.
|
|
||||||
For example variable `S3_GW_PEERS_0_WEIGHT=1` will be transformed to:
|
|
||||||
```
|
|
||||||
peers:
|
|
||||||
0:
|
|
||||||
weight: 1
|
|
||||||
```
|
|
||||||
|
|
||||||
If parameter doesn't support environment variable (e.g. `--listen_address 0.0.0.0:8084`) form it is used as is:
|
|
||||||
```
|
|
||||||
listen_address: 0.0.0.0:8084
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Cache parameters
|
|
||||||
|
|
||||||
Parameters for caches in s3-gw can be specified in .yaml config file. E.g.:
|
|
||||||
```
|
|
||||||
cache:
|
|
||||||
lifetime: 300s
|
|
||||||
size: 150
|
|
||||||
list_objects_lifetime: 1m
|
|
||||||
```
|
|
||||||
If invalid values are set, the gateway will use default values instead.
|
|
||||||
|
|
||||||
## NeoFS AuthMate
|
|
||||||
|
|
||||||
Authmate is a tool to create gateway AWS credentials. AWS users
|
|
||||||
are authenticated with access key IDs and secrets, while NeoFS users are
|
|
||||||
authenticated with key pairs. To complicate things further we have S3 gateway
|
|
||||||
that usually acts on behalf of some user, but user doesn't necessarily want to
|
|
||||||
give his keys to the gateway.
|
|
||||||
|
|
||||||
To solve this we use NeoFS bearer tokens that are signed by the owner (NeoFS
|
|
||||||
"user") and that can implement any kind of policy for NeoFS requests allowed
|
|
||||||
using this token. But tokens can't be used directly as AWS credentials, thus
|
|
||||||
they're stored on NeoFS as regular objects and access key ID is just an
|
|
||||||
address of this object while secret is generated randomly.
|
|
||||||
|
|
||||||
Tokens are not stored on NeoFS in plaintext, they're encrypted with a set of
|
|
||||||
gateway keys. So in order for gateway to be able to successfully extract bearer
|
|
||||||
token the object needs to be stored in a container available for the gateway
|
|
||||||
to read and it needs to be encrypted with this gateway's key (among others
|
|
||||||
potentially).
|
|
||||||
|
|
||||||
### Variables
|
|
||||||
Authmate support the following variables to decrypt wallets provided by `--wallet` and `--gate-wallet`
|
|
||||||
parameters respectevely:
|
|
||||||
* `AUTHMATE_WALLET_PASSPHRASE`
|
|
||||||
* `AUTHMATE_WALLET_GATE_PASSPHRASE`
|
|
||||||
|
|
||||||
If the passphrase is not specified, you will be asked to enter the password interactively:
|
|
||||||
```
|
|
||||||
Enter password for wallet.json >
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Generation of wallet
|
|
||||||
|
|
||||||
To generate wallets for gateways, run the following command:
|
|
||||||
|
|
||||||
```
|
|
||||||
$ ./neo-go wallet init -a -w wallet.json
|
|
||||||
|
|
||||||
Enter the name of the account > AccountTestName
|
|
||||||
Enter passphrase >
|
|
||||||
Confirm passphrase >
|
|
||||||
|
|
||||||
{
|
|
||||||
"version": "3.0",
|
|
||||||
"accounts": [
|
|
||||||
{
|
|
||||||
"address": "NhLQpDnerpviUWDF77j5qyjFgavCmasJ4p",
|
|
||||||
"key": "6PYUFyYpJ1JGyMrYV8NqeUFLKfpEVHsGGjCYtTDkjnKaSgYizRBZxVerte",
|
|
||||||
"label": "AccountTestName",
|
|
||||||
"contract": {
|
|
||||||
"script": "DCECXCsUZPwUyKHs6nAyyCvJ5s/vLwZkkVtWNC0zWzH8a9dBVuezJw==",
|
|
||||||
"parameters": [
|
|
||||||
{
|
|
||||||
"name": "parameter0",
|
|
||||||
"type": "Signature"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"deployed": false
|
|
||||||
},
|
|
||||||
"lock": false,
|
|
||||||
"isDefault": false
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"scrypt": {
|
|
||||||
"n": 16384,
|
|
||||||
"r": 8,
|
|
||||||
"p": 8
|
|
||||||
},
|
|
||||||
"extra": {
|
|
||||||
"Tokens": null
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
wallet successfully created, file location is wallet.json
|
|
||||||
```
|
|
||||||
|
|
||||||
To get public key from wallet run:
|
|
||||||
```
|
|
||||||
$ ./bin/neo-go wallet dump-keys -w wallet.json
|
|
||||||
|
|
||||||
NhLQpDnerpviUWDF77j5qyjFgavCmasJ4p (simple signature contract):
|
|
||||||
025c2b1464fc14c8a1ecea7032c82bc9e6cfef2f0664915b56342d335b31fc6bd7
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Issuance of a secret
|
|
||||||
|
|
||||||
To issue a secret means to create a Bearer and (optionally) Session tokens and
|
|
||||||
put them as an object into container on the NeoFS network. The tokens are
|
|
||||||
encrypted by a set of gateway keys, so you need to pass them as well.
|
|
||||||
|
|
||||||
If a parameter `container-id` is not set, a new container will be created.
|
|
||||||
|
|
||||||
Creation of the bearer token is mandatory, and creation of the session token is
|
|
||||||
optional. If you want to add the session token you need to add a parameter
|
|
||||||
`create-session-token`.
|
|
||||||
|
|
||||||
Rules for bearer token can be set via param `bearer-rules` (json-string and file path allowed), if it is not set,
|
|
||||||
it will be auto-generated with values:
|
|
||||||
|
|
||||||
```
|
|
||||||
{
|
|
||||||
"version": {
|
|
||||||
"major": 2,
|
|
||||||
"minor": 6
|
|
||||||
},
|
|
||||||
"containerID": {
|
|
||||||
"value": "%CID"
|
|
||||||
},
|
|
||||||
"records": [
|
|
||||||
{
|
|
||||||
"operation": "GET",
|
|
||||||
"action": "ALLOW",
|
|
||||||
"filters": [],
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"role": "OTHERS",
|
|
||||||
"keys": []
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Rules for session token can be set via param `session-rules` (json-string and file path allowed), default value is:
|
|
||||||
```
|
|
||||||
{
|
|
||||||
"verb": "PUT",
|
|
||||||
"wildcard": true,
|
|
||||||
"containerID": null
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
If `session-rules` is set, but `create-session-token` is not, the session
|
|
||||||
token will not be created.
|
|
||||||
|
|
||||||
Rules for mapping of `LocationConstraint` ([aws spec](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html#API_CreateBucket_RequestBody))
|
|
||||||
to `PlacementPolicy` ([neofs spec](https://github.com/nspcc-dev/neofs-spec/blob/master/01-arch/02-policy.md))
|
|
||||||
can be set via param `container-policy` (json-string and file path allowed):
|
|
||||||
```
|
|
||||||
{
|
|
||||||
"rep-3": "REP 3",
|
|
||||||
"complex": "REP 1 IN X CBF 1 SELECT 1 FROM * AS X",
|
|
||||||
"example-json-policy": "{\"replicas\":[{\"count\":3,\"selector\":\"SelASD0\"}],\"container_backup_factor\":3,\"selectors\":[{\"name\":\"SelASD0\",\"count\":3,\"filter\":\"*\"}],\"filters\":[]}"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
Example of a command to issue a secret with custom rules for multiple gates:
|
|
||||||
```
|
|
||||||
$ ./neofs-authmate issue-secret --wallet wallet.json \
|
|
||||||
--peer 192.168.130.71:8080 \
|
|
||||||
--bearer-rules '{"records":[{"operation":"PUT","action":"ALLOW","filters":[],"targets":[{"role":"OTHERS","keys":[]}]}]}' \
|
|
||||||
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \
|
|
||||||
--gate-public-key 0317585fa8274f7afdf1fc5f2a2e7bece549d5175c4e5182e37924f30229aef967 \
|
|
||||||
--create-session-token \
|
|
||||||
--session-rules '{"verb":"DELETE","wildcard":false,"containerID":{"value":"%CID"}}'
|
|
||||||
--container-policy '{"rep-3": "REP 3"}'
|
|
||||||
|
|
||||||
Enter password for wallet.json >
|
|
||||||
{
|
|
||||||
"access_key_id": "5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM",
|
|
||||||
"secret_access_key": "438bbd8243060e1e1c9dd4821756914a6e872ce29bf203b68f81b140ac91231c",
|
|
||||||
"owner_private_key": "274fdd6e71fc6a6b8fe77bec500254115d66d6d17347d7db0880d2eb80afc72a",
|
|
||||||
"container_id":"5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Access key ID and secret access key are AWS credentials that you can use with
|
|
||||||
any S3 client.
|
|
||||||
|
|
||||||
Access key ID consists of Base58 encoded containerID(cid) and objectID(oid) stored on the NeoFS network and containing
|
|
||||||
the secret. Format of access_key_id: `%cid0%oid`, where 0(zero) is a delimiter.
|
|
||||||
|
|
||||||
#### Obtainment of a secret access key
|
|
||||||
|
|
||||||
You can get a secret access key associated with access key ID by obtaining a
|
|
||||||
secret stored on the NeoFS network. Here example of providing one password (for `wallet.json`) via env variable
|
|
||||||
and other (for `gate-wallet.json`) interactively:
|
|
||||||
|
|
||||||
```
|
|
||||||
$ AUTHMATE_WALLET_PASSPHRASE=some-pwd \
|
|
||||||
./neofs-authmate obtain-secret --wallet wallet.json \
|
|
||||||
--peer 192.168.130.71:8080 \
|
|
||||||
--gate-wallet gate-wallet.json \
|
|
||||||
--access-key-id 5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM
|
|
||||||
|
|
||||||
Enter password for gate-wallet.json >
|
|
||||||
{
|
|
||||||
"secret_access_key": "438bbd8243060e1e1c9dd4821756914a6e872ce29bf203b68f81b140ac91231c"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## AWS CLI usage
|
|
||||||
|
|
||||||
### Configuration
|
|
||||||
|
|
||||||
#### Credentials
|
|
||||||
|
|
||||||
To configure basic settings that the AWS CLI uses to interact with the Gateway, do the following steps:
|
|
||||||
|
|
||||||
1. issue a secret with neofs-authmate tool (see [NeoFS Authmate] (#neofs-authmate))
|
|
||||||
2. execute the command
|
|
||||||
```
|
|
||||||
$ aws configure
|
|
||||||
```
|
|
||||||
after you enter this command, the AWS CLI will prompt you for four pieces of information, like in this example
|
|
||||||
(replace with your own values):
|
|
||||||
```
|
|
||||||
AWS Access Key ID [None]: 5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM
|
|
||||||
AWS Secret Access Key [None]: 438bbd8243060e1e1c9dd4821756914a6e872ce29bf203b68f81b140ac91231c
|
|
||||||
Default region name [None]: ru
|
|
||||||
Default output format [none]: json
|
|
||||||
```
|
|
||||||
|
|
||||||
### Basic usage
|
|
||||||
|
|
||||||
> **_NOTE:_** To specify IP and port of the gate, append `--endpoint-url https://%IP:%PORT` to your commands.
|
|
||||||
|
|
||||||
#### Bucket
|
|
||||||
|
|
||||||
##### Obtainment of a list of buckets
|
|
||||||
|
|
||||||
To view the list of the buckets in the NeoFS node, to which the gateway is connected, enter the command:
|
|
||||||
```
|
|
||||||
$ aws s3 ls
|
|
||||||
```
|
|
||||||
|
|
||||||
##### Creation of a bucket
|
|
||||||
|
|
||||||
At this moment, the gateway supports only canned ACL and doesn't support the setting of location constraints.
|
|
||||||
|
|
||||||
To create a bucket, run the command:
|
|
||||||
```
|
|
||||||
$ aws s3api create-bucket --bucket %BUCKET_NAME --acl %ACL
|
|
||||||
```
|
|
||||||
where `%ACL` can be represented by a hex encoded value or by keywords `public-read-write`, `private`, `public-read`.
|
|
||||||
If the parameter is not set, the default value is `private`.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
##### Deletion of a bucket
|
|
||||||
|
|
||||||
To delete a bucket, execute the following command:
|
|
||||||
```
|
|
||||||
$ aws s3api delete-bucket --bucket %BUCKET_NAME
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Object
|
|
||||||
|
|
||||||
##### Obtainment of a list of objects
|
|
||||||
|
|
||||||
To view the list of the objects in a bucket, run:
|
|
||||||
```
|
|
||||||
$ aws s3api list-objects --bucket %BUCKET_NAME
|
|
||||||
```
|
|
||||||
|
|
||||||
##### Upload of a file
|
|
||||||
|
|
||||||
To upload the file into a bucket in the NeoFS network, run the following command:
|
|
||||||
```
|
|
||||||
$ aws s3api put-object --bucket %BUCKET_NAME --key %OBJECT_KEY --body %FILEPATH
|
|
||||||
```
|
|
||||||
where %OBJECT_KEY is a filename of an object in NeoFS
|
|
||||||
|
|
||||||
##### Upload of a dir
|
|
||||||
|
|
||||||
To upload the dir into a bucket in the NeoFS network, run the following command:
|
|
||||||
|
|
||||||
```
|
|
||||||
$ aws s3 sync %DIRPATH s3://%BUCKET_NAME
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Download of a file
|
|
||||||
|
|
||||||
To download the file from a bucket in the NeoFS Network, execute:
|
|
||||||
```
|
|
||||||
$ aws s3api get-object --bucket %BUCKET_NAME --key %OBJECT_KEY %OUTFILE
|
|
||||||
```
|
|
||||||
|
|
||||||
where %OUTFILE is a file to store object content.
|
|
||||||
|
|
||||||
#### Deletion of a file
|
|
||||||
To delete the file:
|
|
||||||
```
|
|
||||||
$ aws s3api delete-object --bucket %BUCKET_NAME --key %FILE_NAME
|
|
||||||
```
|
|
||||||
|
|
||||||
## S3 API supported
|
## S3 API supported
|
||||||
|
|
||||||
|
|
191
docs/authmate.md
Normal file
191
docs/authmate.md
Normal file
|
@ -0,0 +1,191 @@
|
||||||
|
# NeoFS AuthMate
|
||||||
|
|
||||||
|
Authmate is a tool to create gateway AWS credentials. AWS users
|
||||||
|
are authenticated with access key IDs and secrets, while NeoFS users are
|
||||||
|
authenticated with key pairs. To complicate things further we have S3 gateway
|
||||||
|
that usually acts on behalf of some user, but user doesn't necessarily want to
|
||||||
|
give his keys to the gateway.
|
||||||
|
|
||||||
|
To solve this we use NeoFS bearer tokens that are signed by the owner (NeoFS
|
||||||
|
"user") and that can implement any kind of policy for NeoFS requests allowed
|
||||||
|
using this token. But tokens can't be used directly as AWS credentials, thus
|
||||||
|
they're stored on NeoFS as regular objects and access key ID is just an
|
||||||
|
address of this object while secret is generated randomly.
|
||||||
|
|
||||||
|
Tokens are not stored on NeoFS in plaintext, they're encrypted with a set of
|
||||||
|
gateway keys. So in order for gateway to be able to successfully extract bearer
|
||||||
|
token the object needs to be stored in a container available for the gateway
|
||||||
|
to read and it needs to be encrypted with this gateway's key (among others
|
||||||
|
potentially).
|
||||||
|
|
||||||
|
## Variables
|
||||||
|
Authmate support the following variables to decrypt wallets provided by `--wallet` and `--gate-wallet`
|
||||||
|
parameters respectevely:
|
||||||
|
* `AUTHMATE_WALLET_PASSPHRASE`
|
||||||
|
* `AUTHMATE_WALLET_GATE_PASSPHRASE`
|
||||||
|
|
||||||
|
If the passphrase is not specified, you will be asked to enter the password interactively:
|
||||||
|
```
|
||||||
|
Enter password for wallet.json >
|
||||||
|
```
|
||||||
|
|
||||||
|
## Generation of wallet
|
||||||
|
|
||||||
|
To generate wallets for gateways, run the following command:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ./neo-go wallet init -a -w wallet.json
|
||||||
|
|
||||||
|
Enter the name of the account > AccountTestName
|
||||||
|
Enter passphrase >
|
||||||
|
Confirm passphrase >
|
||||||
|
|
||||||
|
{
|
||||||
|
"version": "3.0",
|
||||||
|
"accounts": [
|
||||||
|
{
|
||||||
|
"address": "NhLQpDnerpviUWDF77j5qyjFgavCmasJ4p",
|
||||||
|
"key": "6PYUFyYpJ1JGyMrYV8NqeUFLKfpEVHsGGjCYtTDkjnKaSgYizRBZxVerte",
|
||||||
|
"label": "AccountTestName",
|
||||||
|
"contract": {
|
||||||
|
"script": "DCECXCsUZPwUyKHs6nAyyCvJ5s/vLwZkkVtWNC0zWzH8a9dBVuezJw==",
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "parameter0",
|
||||||
|
"type": "Signature"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"deployed": false
|
||||||
|
},
|
||||||
|
"lock": false,
|
||||||
|
"isDefault": false
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"scrypt": {
|
||||||
|
"n": 16384,
|
||||||
|
"r": 8,
|
||||||
|
"p": 8
|
||||||
|
},
|
||||||
|
"extra": {
|
||||||
|
"Tokens": null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
wallet successfully created, file location is wallet.json
|
||||||
|
```
|
||||||
|
|
||||||
|
To get public key from wallet run:
|
||||||
|
```
|
||||||
|
$ ./bin/neo-go wallet dump-keys -w wallet.json
|
||||||
|
|
||||||
|
NhLQpDnerpviUWDF77j5qyjFgavCmasJ4p (simple signature contract):
|
||||||
|
025c2b1464fc14c8a1ecea7032c82bc9e6cfef2f0664915b56342d335b31fc6bd7
|
||||||
|
```
|
||||||
|
|
||||||
|
## Issuance of a secret
|
||||||
|
|
||||||
|
To issue a secret means to create a Bearer and (optionally) Session tokens and
|
||||||
|
put them as an object into container on the NeoFS network. The tokens are
|
||||||
|
encrypted by a set of gateway keys, so you need to pass them as well.
|
||||||
|
|
||||||
|
If a parameter `container-id` is not set, a new container will be created.
|
||||||
|
|
||||||
|
Creation of the bearer token is mandatory, and creation of the session token is
|
||||||
|
optional. If you want to add the session token you need to add a parameter
|
||||||
|
`create-session-token`.
|
||||||
|
|
||||||
|
Rules for bearer token can be set via param `bearer-rules` (json-string and file path allowed), if it is not set,
|
||||||
|
it will be auto-generated with values:
|
||||||
|
|
||||||
|
```
|
||||||
|
{
|
||||||
|
"version": {
|
||||||
|
"major": 2,
|
||||||
|
"minor": 6
|
||||||
|
},
|
||||||
|
"containerID": {
|
||||||
|
"value": "%CID"
|
||||||
|
},
|
||||||
|
"records": [
|
||||||
|
{
|
||||||
|
"operation": "GET",
|
||||||
|
"action": "ALLOW",
|
||||||
|
"filters": [],
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"role": "OTHERS",
|
||||||
|
"keys": []
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Rules for session token can be set via param `session-rules` (json-string and file path allowed), default value is:
|
||||||
|
```
|
||||||
|
{
|
||||||
|
"verb": "PUT",
|
||||||
|
"wildcard": true,
|
||||||
|
"containerID": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
If `session-rules` is set, but `create-session-token` is not, the session
|
||||||
|
token will not be created.
|
||||||
|
|
||||||
|
Rules for mapping of `LocationConstraint` ([aws spec](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html#API_CreateBucket_RequestBody))
|
||||||
|
to `PlacementPolicy` ([neofs spec](https://github.com/nspcc-dev/neofs-spec/blob/master/01-arch/02-policy.md))
|
||||||
|
can be set via param `container-policy` (json-string and file path allowed):
|
||||||
|
```
|
||||||
|
{
|
||||||
|
"rep-3": "REP 3",
|
||||||
|
"complex": "REP 1 IN X CBF 1 SELECT 1 FROM * AS X",
|
||||||
|
"example-json-policy": "{\"replicas\":[{\"count\":3,\"selector\":\"SelASD0\"}],\"container_backup_factor\":3,\"selectors\":[{\"name\":\"SelASD0\",\"count\":3,\"filter\":\"*\"}],\"filters\":[]}"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Example of a command to issue a secret with custom rules for multiple gates:
|
||||||
|
```
|
||||||
|
$ ./neofs-authmate issue-secret --wallet wallet.json \
|
||||||
|
--peer 192.168.130.71:8080 \
|
||||||
|
--bearer-rules '{"records":[{"operation":"PUT","action":"ALLOW","filters":[],"targets":[{"role":"OTHERS","keys":[]}]}]}' \
|
||||||
|
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \
|
||||||
|
--gate-public-key 0317585fa8274f7afdf1fc5f2a2e7bece549d5175c4e5182e37924f30229aef967 \
|
||||||
|
--create-session-token \
|
||||||
|
--session-rules '{"verb":"DELETE","wildcard":false,"containerID":{"value":"%CID"}}'
|
||||||
|
--container-policy '{"rep-3": "REP 3"}'
|
||||||
|
|
||||||
|
Enter password for wallet.json >
|
||||||
|
{
|
||||||
|
"access_key_id": "5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM",
|
||||||
|
"secret_access_key": "438bbd8243060e1e1c9dd4821756914a6e872ce29bf203b68f81b140ac91231c",
|
||||||
|
"owner_private_key": "274fdd6e71fc6a6b8fe77bec500254115d66d6d17347d7db0880d2eb80afc72a",
|
||||||
|
"container_id":"5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Access key ID and secret access key are AWS credentials that you can use with
|
||||||
|
any S3 client.
|
||||||
|
|
||||||
|
Access key ID consists of Base58 encoded containerID(cid) and objectID(oid) stored on the NeoFS network and containing
|
||||||
|
the secret. Format of access_key_id: `%cid0%oid`, where 0(zero) is a delimiter.
|
||||||
|
|
||||||
|
## Obtainment of a secret access key
|
||||||
|
|
||||||
|
You can get a secret access key associated with access key ID by obtaining a
|
||||||
|
secret stored on the NeoFS network. Here example of providing one password (for `wallet.json`) via env variable
|
||||||
|
and other (for `gate-wallet.json`) interactively:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ AUTHMATE_WALLET_PASSPHRASE=some-pwd \
|
||||||
|
./neofs-authmate obtain-secret --wallet wallet.json \
|
||||||
|
--peer 192.168.130.71:8080 \
|
||||||
|
--gate-wallet gate-wallet.json \
|
||||||
|
--access-key-id 5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM
|
||||||
|
|
||||||
|
Enter password for gate-wallet.json >
|
||||||
|
{
|
||||||
|
"secret_access_key": "438bbd8243060e1e1c9dd4821756914a6e872ce29bf203b68f81b140ac91231c"
|
||||||
|
}
|
||||||
|
```
|
92
docs/aws_cli.md
Normal file
92
docs/aws_cli.md
Normal file
|
@ -0,0 +1,92 @@
|
||||||
|
# AWS CLI basic usage
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Credentials
|
||||||
|
|
||||||
|
To configure basic settings that the AWS CLI uses to interact with the Gateway, do the following steps:
|
||||||
|
|
||||||
|
1. issue a secret with neofs-authmate tool (see [NeoFS Authmate] (#neofs-authmate))
|
||||||
|
2. execute the command
|
||||||
|
```
|
||||||
|
$ aws configure
|
||||||
|
```
|
||||||
|
after you enter this command, the AWS CLI will prompt you for four pieces of information, like in this example
|
||||||
|
(replace with your own values):
|
||||||
|
```
|
||||||
|
AWS Access Key ID [None]: 5g933dyLEkXbbAspouhPPTiyLZRg4axBW1axSPD87eVT0AiXsH4AjYy1iTJ4C1WExzjBrSobJsQFWEyKLREe5sQYM
|
||||||
|
AWS Secret Access Key [None]: 438bbd8243060e1e1c9dd4821756914a6e872ce29bf203b68f81b140ac91231c
|
||||||
|
Default region name [None]: ru
|
||||||
|
Default output format [none]: json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Basic usage
|
||||||
|
|
||||||
|
> **_NOTE:_** To specify IP and port of the gate, append `--endpoint-url https://%IP:%PORT` to your commands.
|
||||||
|
|
||||||
|
### Bucket
|
||||||
|
|
||||||
|
#### Obtainment of a list of buckets
|
||||||
|
|
||||||
|
To view the list of the buckets in the NeoFS node, to which the gateway is connected, enter the command:
|
||||||
|
```
|
||||||
|
$ aws s3 ls
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Creation of a bucket
|
||||||
|
|
||||||
|
At this moment, the gateway supports only canned ACL and doesn't support the setting of location constraints.
|
||||||
|
|
||||||
|
To create a bucket, run the command:
|
||||||
|
```
|
||||||
|
$ aws s3api create-bucket --bucket %BUCKET_NAME --acl %ACL
|
||||||
|
```
|
||||||
|
where `%ACL` can be represented by a hex encoded value or by keywords `public-read-write`, `private`, `public-read`.
|
||||||
|
If the parameter is not set, the default value is `private`.
|
||||||
|
|
||||||
|
#### Deletion of a bucket
|
||||||
|
|
||||||
|
To delete a bucket, execute the following command:
|
||||||
|
```
|
||||||
|
$ aws s3api delete-bucket --bucket %BUCKET_NAME
|
||||||
|
```
|
||||||
|
|
||||||
|
### Object
|
||||||
|
|
||||||
|
#### Obtainment of a list of objects
|
||||||
|
|
||||||
|
To view the list of the objects in a bucket, run:
|
||||||
|
```
|
||||||
|
$ aws s3api list-objects --bucket %BUCKET_NAME
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Upload of a file
|
||||||
|
|
||||||
|
To upload the file into a bucket in the NeoFS network, run the following command:
|
||||||
|
```
|
||||||
|
$ aws s3api put-object --bucket %BUCKET_NAME --key %OBJECT_KEY --body %FILEPATH
|
||||||
|
```
|
||||||
|
where %OBJECT_KEY is a filename of an object in NeoFS
|
||||||
|
|
||||||
|
#### Upload of a dir
|
||||||
|
|
||||||
|
To upload the dir into a bucket in the NeoFS network, run the following command:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ aws s3 sync %DIRPATH s3://%BUCKET_NAME
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Download of a file
|
||||||
|
|
||||||
|
To download the file from a bucket in the NeoFS Network, execute:
|
||||||
|
```
|
||||||
|
$ aws s3api get-object --bucket %BUCKET_NAME --key %OBJECT_KEY %OUTFILE
|
||||||
|
```
|
||||||
|
|
||||||
|
where %OUTFILE is a file to store object content.
|
||||||
|
|
||||||
|
#### Deletion of a file
|
||||||
|
To delete the file:
|
||||||
|
```
|
||||||
|
$ aws s3api delete-object --bucket %BUCKET_NAME --key %FILE_NAME
|
||||||
|
```
|
101
docs/configuration.md
Normal file
101
docs/configuration.md
Normal file
|
@ -0,0 +1,101 @@
|
||||||
|
# Configuration
|
||||||
|
|
||||||
|
In general, everything available as CLI parameter can also be specified via
|
||||||
|
environment variables, so they're not specifically mentioned in most cases
|
||||||
|
(see `--help` also). If you prefer a config file you can use it in yaml format.
|
||||||
|
|
||||||
|
## Nodes and weights
|
||||||
|
|
||||||
|
You can specify multiple `-p` options to add more NeoFS nodes, this will make
|
||||||
|
gateway spread requests equally among them (using weight 1 for every node):
|
||||||
|
|
||||||
|
```
|
||||||
|
$ neofs-s3-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080
|
||||||
|
```
|
||||||
|
If you want some specific load distribution proportions, use weights, but they
|
||||||
|
can only be specified via environment variables:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ HTTP_GW_PEERS_0_ADDRESS=192.168.130.72:8080 HTTP_GW_PEERS_0_WEIGHT=9 \
|
||||||
|
HTTP_GW_PEERS_1_ADDRESS=192.168.130.71:8080 HTTP_GW_PEERS_1_WEIGHT=1 neofs-s3-gw
|
||||||
|
```
|
||||||
|
This command will make gateway use 192.168.130.72 for 90% of requests and
|
||||||
|
192.168.130.71 for remaining 10%.
|
||||||
|
|
||||||
|
## Key
|
||||||
|
|
||||||
|
Wallet (`--wallet`) is mandatory parameter. It is a path to wallet file. You can provide password to decrypt wallet
|
||||||
|
via `S3_GW_WALLET_PASSPHRASE` variable or you will be asked to enter the password interactively.
|
||||||
|
You also can specify account address to use from wallet using `--address` parameter.
|
||||||
|
|
||||||
|
## Binding and TLS
|
||||||
|
|
||||||
|
Gateway binds to `0.0.0.0:8080` by default and you can change that with
|
||||||
|
`--listen_address` option.
|
||||||
|
|
||||||
|
It can also provide TLS interface for its users, just specify paths to key and
|
||||||
|
certificate files via `--tls.key_file` and `--tls.cert_file` parameters. Note
|
||||||
|
that using these options makes gateway TLS-only, if you need to serve both TLS
|
||||||
|
and plain text you either have to run two gateway instances or use some
|
||||||
|
external redirecting solution.
|
||||||
|
|
||||||
|
Example to bind to `192.168.130.130:443` and serve TLS there (keys and nodes
|
||||||
|
omitted):
|
||||||
|
|
||||||
|
```
|
||||||
|
$ neofs-s3-gw --listen_address 192.168.130.130:443 \
|
||||||
|
--tls.key_file=key.pem --tls.cert_file=cert.pem
|
||||||
|
```
|
||||||
|
|
||||||
|
## Monitoring and metrics
|
||||||
|
|
||||||
|
Pprof and Prometheus are integrated into the gateway, but not enabled by
|
||||||
|
default. To enable them use `--pprof` and `--metrics` flags or
|
||||||
|
`HTTP_GW_PPROF`/`HTTP_GW_METRICS` environment variables.
|
||||||
|
|
||||||
|
## Yaml file
|
||||||
|
Configuration file is optional and can be used instead of environment variables/other parameters.
|
||||||
|
It can be specified with `--config` parameter:
|
||||||
|
```
|
||||||
|
$ neofs-s3-gw --config your-config.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Configuration file example:
|
||||||
|
```
|
||||||
|
listen_address: 0.0.0.0:8084
|
||||||
|
|
||||||
|
wallet:
|
||||||
|
passphrase: 123456
|
||||||
|
|
||||||
|
logger:
|
||||||
|
level: debug
|
||||||
|
|
||||||
|
peers:
|
||||||
|
0:
|
||||||
|
address: s01.neofs.devenv:8080
|
||||||
|
weight: 1
|
||||||
|
```
|
||||||
|
|
||||||
|
To know nesting level of variable you need to cut off the prefix `S3_GW` from variable and split the rest parts by `_`.
|
||||||
|
For example variable `S3_GW_PEERS_0_WEIGHT=1` will be transformed to:
|
||||||
|
```
|
||||||
|
peers:
|
||||||
|
0:
|
||||||
|
weight: 1
|
||||||
|
```
|
||||||
|
|
||||||
|
If parameter doesn't support environment variable (e.g. `--listen_address 0.0.0.0:8084`) form it is used as is:
|
||||||
|
```
|
||||||
|
listen_address: 0.0.0.0:8084
|
||||||
|
```
|
||||||
|
|
||||||
|
### Cache parameters
|
||||||
|
|
||||||
|
Parameters for caches in s3-gw can be specified in .yaml config file. E.g.:
|
||||||
|
```
|
||||||
|
cache:
|
||||||
|
lifetime: 300s
|
||||||
|
size: 150
|
||||||
|
list_objects_lifetime: 1m
|
||||||
|
```
|
||||||
|
If invalid values are set, the gateway will use default values instead.
|
Loading…
Reference in a new issue