frostfs-s3-gw/docs/configuration.md

Configuration

There are three ways to configure the S3 GW:

1. CLI parameters
2. YAML file
3. Environment variables

Everything available as a CLI parameter can also be specified via environment variables and almost everything can be specified via .yaml configuration file.

But not vice versa, some parameters can be configured only with environment variables/configuration file. Most of these parameters have default values, therefore, these ways to configure the gateway are optional and basic configuration can be completed with CLI parameters only.

1. CLI parameters
   1. Nodes and weights
   2. Wallet
   3. Binding and TLS
   4. RPC endpoint and resolving of bucket names
   5. Processing of requests
   6. Connection to FrostFS
   7. Monitoring and metrics
2. YAML file and environment variables
   1. Configuration file

CLI parameters

Nodes and weights

You can specify multiple -p options to add more FrostFS nodes; this will make a gateway spread requests equally among them (using weight 1 for every node):

$ frostfs-s3-gw -p 192.168.130.72:8080 -p 192.168.130.71:8080

If you want some specific load distribution proportions, use weights and priorities, they can only be specified via environment variables or a configuration file.

Wallet

Wallet (--wallet) is a mandatory parameter. It is a path to a wallet file. You can provide a passphrase to decrypt a wallet via env variable or conf file, or you will be asked to enter a password interactively. You can also specify an account address to use from a wallet using the --address parameter.

Listening on address and TLS

You can make the gateway listen on specific address using the --listen_address option.

It can also provide TLS interface for its users, just specify paths to the key and certificate files via --tls.key_file and --tls.cert_file parameters. Note that using these options makes gateway TLS-only. If you need to serve both TLS and plain text, you either have to run two gateway instances or use some external redirecting solution.

Example to bind to 192.168.130.130:443 and serve TLS there (keys and nodes are omitted):

$ frostfs-s3-gw --listen_address 192.168.130.130:443 \
  --tls.key_file=key.pem --tls.cert_file=cert.pem

Using these flag you can configure only one address. To set multiple addresses use yaml config.

RPC endpoint and resolving of bucket names

To set RPC endpoint specify a value of parameter -r or --rpc_endpoint. The parameter is required if another parameter's --resolve_order value contains nns.

$ frostfs-s3-gw --rpc_endpoint http://morph-chain.frostfs.devenv:30333/ --resolve_order nns,dns

Processing of requests

Maximum number of clients whose requests can be handled by the gateway can be specified by the value of --max_clients_count parameter. --max_clients_deadline defines deadline after which the gate sends error RequestTimeout to a client.

$ frostfs-s3-gw --max_clients_count 150 --max_clients_deadline 1m

Connection to FrostFS

Timeout to connect to FrostFS nodes can be set with --connect_timeout and timeout to check node health during rebalance--healthcheck_timeout.

Also, interval to check node health can be specified by --rebalance_interval value.

$ frostfs-s3-gw --healthcheck_timeout 15s --connect_timeout 1m --rebalance_interval 1h

Monitoring and metrics

Pprof and Prometheus are integrated into the gateway. To enable them, use --pprof and --metrics flags or S3_GW_PPROF_ENABLED/S3_GW_PROMETHEUS_ENABLED environment variables.

YAML file and environment variables

Example of a YAML configuration file: yaml-example Examples of environment variables: env-example.

A path to a configuration file can be specified with --config parameter:

$ frostfs-s3-gw --config your-config.yaml

Multiple configs

You can use several config files when running application. It allows you to split configuration into parts. For example, you can use separate yaml file for pprof and prometheus section in config (see config examples). You can either provide several files with repeating --config flag or provide path to the dir that contains all configs using --config-dir flag. Also, you can combine these flags:

$ frostfs-s3-gw --config ./config/config.yaml --config /your/partial/config.yaml --config-dir ./config/dir

Note: next file in --config flag overwrites values from the previous one. Files from --config-dir directory overwrite values from --config files. So the command above run frostfs-s3-gw to listen on 0.0.0.0:8080 address (value from ./config/config.yaml), applies parameters from /your/partial/config.yaml, enable pprof (value from ./config/dir/pprof.yaml) and prometheus (value from ./config/dir/prometheus.yaml).

Reload on SIGHUP

Some config values can be reloaded on SIGHUP signal. Such parameters have special mark in tables below.

You can send SIGHUP signal to app using the following command:

$ kill -s SIGHUP <app_pid>

Example:

$ ./bin/frostfs-s3-gw --config config.yaml  &> s3.log &
[1] 998346

$ cat s3.log
# ...
2022-09-30T17:38:22.338+0300    info    s3-gw/app.go:371        application started     {"name": "frostfs-s3-gw", "version": "v0.24.0"}
# ...

$ kill -s SIGHUP 998346

$ cat s3.log
# ...
2022-09-30T17:38:40.909+0300    info    s3-gw/app.go:491        SIGHUP config reload completed

FrostFS S3 Gateway configuration file

This section contains detailed FrostFS S3 Gateway configuration file description including default config values and some tips to set up configurable values.

There are some custom types used for brevity:

• duration -- string consisting of a number and a suffix. Suffix examples include s (seconds), m (minutes), ms ( milliseconds).

Structure

Section	Description
no section	General parameters
wallet	Wallet configuration
peers	Nodes configuration
placement_policy	Placement policy configuration
server	Server configuration
logger	Logger configuration
http_logging	HTTP Request logger configuration
cache	Cache configuration
cors	CORS configuration
pprof	Pprof configuration
prometheus	Prometheus configuration
tracing	Tracing configuration
frostfs	Parameters of requests to FrostFS
resolve_bucket	Bucket name resolving configuration
kludge	Different kludge configuration
runtime	Runtime configuration
features	Features configuration
web	Web server configuration
frostfsid	FrostfsID configuration
policy	Policy contract configuration
proxy	Proxy contract configuration
namespaces	Namespaces configuration
retry	Retry configuration
containers	Containers configuration
vhs	VHS configuration

General section

listen_domains:
   - s3dev.frostfs.devenv
   - s3dev.<wildcard>.frostfs.devenv
   - s3dev2.frostfs.devenv

rpc_endpoint: http://morph-chain.frostfs.devenv:30333
resolve_order:
  - nns
  - dns

connect_timeout: 10s
stream_timeout: 10s
healthcheck_timeout: 15s
rebalance_interval: 60s
pool_error_threshold: 100

max_clients_count: 100
max_clients_deadline: 30s

allowed_access_key_id_prefixes:
   - Ck9BHsgKcnwfCTUSFm6pxhoNS4cBqgN2NQ8zVgPjqZDX
   - 3stjWenX15YwYzczMr88gy3CQr4NYFBQ8P7keGzH5QFn

reconnect_interval: 1m

source_ip_header: "Source-Ip"

Parameter	Type	SIGHUP reload	Default value	Description
listen_domains	[]string	yes		Domains to be able to use virtual-hosted-style access to bucket. The presence of placeholders of the type is supported.
rpc_endpoint	string	no		The address of the RPC host to which the gateway connects to resolve bucket names and interact with frostfs contracts (required to use the nns resolver and frostfsid contract).
resolve_order	[]string	yes	[dns]	Order of bucket name resolvers to use. Available resolvers: dns, nns.
connect_timeout	duration	no	10s	Timeout to connect to a node.
stream_timeout	duration	no	10s	Timeout for individual operations in streaming RPC.
healthcheck_timeout	duration	no	15s	Timeout to check node health during rebalance.
rebalance_interval	duration	no	60s	Interval to check node health.
pool_error_threshold	uint32	no	100	The number of errors on connection after which node is considered as unhealthy.
max_clients_count	int	no	100	Limits for processing of clients' requests.
max_clients_deadline	duration	no	30s	Deadline after which the gate sends error RequestTimeout to a client.
allowed_access_key_id_prefixes	[]string	no		List of allowed AccessKeyID prefixes which S3 GW serve. If the parameter is omitted, all AccessKeyID will be accepted.
reconnect_interval	duration	no	1m	Listeners reconnection interval.
source_ip_header	string	yes		Custom header to retrieve Source IP.

wallet section

wallet:
   path: /path/to/wallet.json # Path to wallet
   passphrase: "" # Passphrase to decrypt wallet.
   address: NfgHwwTi3wHAS8aFAN243C5vGbkYDpqLHP

Parameter	Type	Default value	Description
path	string		Path to wallet
passphrase	string		Passphrase to decrypt wallet.
address	string		Account address to get from wallet. If omitted default one will be used.

peers section

# Nodes configuration
# This configuration makes the gateway use the first node (node1.frostfs:8080)
# while it's healthy. Otherwise, gateway uses the second node (node2.frostfs:8080)
# for 10% of requests and the third node (node3.frostfs:8080) for 90% of requests.
# Until nodes with the same priority level are healthy
# nodes with other priority are not used.
# The lower the value, the higher the priority.
peers:
  0:
    address: node1.frostfs:8080
    priority: 1
    weight: 1
  1:
    address: node2.frostfs:8080
    priority: 2
    weight: 0.1
  2:
    address: node3.frostfs:8080
    priority: 2
    weight: 0.9

Parameter	Type	Default value	Description
address	string		Address of storage node.
priority	int	1	It allows to group nodes and don't switch group until all nodes with the same priority will be unhealthy. The lower the value, the higher the priority.
weight	float	1	Weight of node in the group with the same priority. Distribute requests to nodes proportionally to these values.

placement_policy section

placement_policy:
  default: REP 3
  region_mapping: /path/to/mapping/rules.json
  copies_numbers:
    - location_constraint: one-dc
      vector:
        - 1
        - 2
        - 3

Parameter	Type	SIGHUP reload	Default value	Description
default	string	yes	REP 3	Default policy of placing containers in FrostFS. If a user sends a request CreateBucket and doesn't define policy for placing of a container in FrostFS, the S3 Gateway will put the container with default policy.
region_mapping	string	yes		Path to file that maps aws LocationContraint values to FrostFS placement policy. The similar to --container-policy flag in frostfs-s3-authmate util, see in docs
copies_numbers	[]Copies numbers	no		Array of configured location constraints and their copies numbers.

File for region_mapping must contain something like this:

{
   "rep-3": "REP 3",
   "complex": "REP 1 IN X CBF 1 SELECT 1 FROM * AS X",
   "example-json-policy": "{\"replicas\":[{\"count\":3,\"selector\":\"SelASD0\"}],\"container_backup_factor\":3,\"selectors\":[{\"name\":\"SelASD0\",\"count\":3,\"filter\":\"*\"}],\"filters\":[]}"
}

Note: on SIGHUP reload policies will be updated only if both parameters are valid. So if you change default to some valid value and set invalid path in region_mapping the default value won't be changed.

copies_numbers subsection

- location_constraint: sample-01
  vector:
    - 1
    - 2
    - 3

Parameter	Type	SIGHUP reload	Default value	Description
location_constraint	string	no		Location constraint text label.
vector	[]int	no		Array of copies numbers corresponding to the constraint.

server section

You can specify several listeners for server. For example, for http and https.

server:
  - address: 0.0.0.0:8080
    tls:
      enabled: false
      cert_file: /path/to/cert
      key_file: /path/to/key
  - address: 0.0.0.0:8081
    tls:
      enabled: true
      cert_file: /path/to/another/cert
      key_file: /path/to/another/key

Parameter	Type	SIGHUP reload	Default value	Description
address	string		0.0.0.0:8080	The address that the gateway is listening on.
tls.enabled	bool		false	Enable TLS or not.
tls.cert_file	string	yes		Path to the TLS certificate.
tls.key_file	string	yes		Path to the key.

logger section

logger:
  level: debug
  destination: stdout

Parameter	Type	SIGHUP reload	Default value	Description
level	string	yes	debug	Logging level. Possible values: debug, info, warn, error, dpanic, panic, fatal.
destination	string	no	stdout	Destination for logger: stdout or journald

http_logging section

Could be enabled only in builds with loghttp build tag. To build with loghttp tag, pass GOFLAGS var to make:

make GOFLAGS="-tags=loghttp" [target]

http_logging:
  enabled: false
  max_body: 1024
  max_log_size: 20
  gzip: true
  destination: stdout

Parameter	Type	SIGHUP reload	Default value	Description
enabled	bool	yes	false	Flag to enable the logger.
max_body	int	yes	1024	Max body size for log output in bytes.
max_log_size	int	yes	50	Log file size threshold (in megabytes) to be moved in backup file. After reaching threshold, initial filename is appended with timestamp. And new empty file with initial name is created.
gzip	bool	yes	false	Whether to enable Gzip compression to backup log files.
destination	string	yes	stdout	Specify path for log output. Accepts log file path, or "stdout" and "stderr" reserved words to print in output streams. File and folders are created if necessary.

cache section

cache:
  objects:
    lifetime: 300s
    size: 150
  list:
    lifetime: 1m
    size: 100
  list_session:
    lifetime: 1m
    size: 100
  names:
    lifetime: 1m
    size: 1000
  buckets:
    lifetime: 1m
    size: 500
  system:
    lifetime: 2m
    size: 1000
  accessbox:
    removing_check_interval: 5m
    lifetime: 10m
    size: 100
  accesscontrol:
    lifetime: 1m
    size: 100000
  morph_policy:
    lifetime: 30s
    size: 10000
  frostfsid:
    lifetime: 1m
    size: 10000
  network_info:
    lifetime: 1m

Parameter	Type	Default value	Description
objects	Cache config	lifetime: 5m size: 1000000	Cache for objects (FrostFS headers).
list	Cache config	lifetime: 60s size: 100000	Cache which keeps lists of objects in buckets.
list_session	Cache config	lifetime: 60s size: 100	Cache which keeps listing session.
names	Cache config	lifetime: 60s size: 10000	Cache which contains mapping of nice name to object addresses.
buckets	Cache config	lifetime: 60s size: 1000	Cache which contains mapping of bucket name to bucket info.
system	Cache config	lifetime: 5m size: 10000	Cache for system objects in a bucket: bucket settings etc.
accessbox	Accessbox cache config	lifetime: 10m size: 100	Cache which stores access box with tokens by its address.
accesscontrol	Cache config	lifetime: 1m size: 100000	Cache which stores owner to cache operation mapping.
morph_policy	Cache config	lifetime: 1m size: 10000	Cache which stores list of policy chains.
frostfsid	Cache config	lifetime: 1m size: 10000	Cache which stores FrostfsID subject info.
network_info	Cache config	lifetime: 1m	Cache which stores network info.

cache subsection

lifetime: 2m
size: 1000

Parameter	Type	Default value	Description
lifetime	duration	depends on cache	Lifetime of entries in cache.
size	int	depends on cache	LRU cache size.

accessbox subsection

lifetime: 10m
size: 100

Parameter	Type	Default value	Description
removing_check_interval	duration	`5m'	Time after which creds should be checked for removal.
lifetime	duration	'10m'	Lifetime of entries in cache.
size	int	'100	LRU cache size.

cors section

cors:
  default_max_age: 600

Parameter	Type	Default value	Description
default_max_age	int	600	Value of Access-Control-Max-Age header in seconds.

pprof section

Contains configuration for the pprof profiler.

pprof:
  enabled: true
  address: localhost:8085

Parameter	Type	SIGHUP reload	Default value	Description
enabled	bool	yes	false	Flag to enable the service.
address	string	yes	localhost:8085	Address that service listener binds to.

prometheus section

Contains configuration for the prometheus metrics service. General metrics are available on /metrics url path, billing metrics on /metrics/billing.

prometheus:
  enabled: true
  address: localhost:8086

Parameter	Type	SIGHUP reload	Default value	Description
enabled	bool	yes	false	Flag to enable the service.
address	string	yes	localhost:8086	Address that service listener binds to.

tracing section

Contains configuration for the tracing service.

tracing:
  enabled: false
  exporter: "otlp_grpc"
  endpoint: "localhost:4318"
  trusted_ca: "/etc/ssl/telemetry-trusted-ca.pem"

Parameter	Type	SIGHUP reload	Default value	Description
enabled	bool	yes	false	Flag to enable the service.
exporter	string	yes	``	Type of tracing exporter.
endpoint	string	yes	``	Address that service listener binds to.
trusted_ca	string	yes		Path to certificate of a certification authority in pem format, that issued the TLS certificate of the telemetry remote server.

frostfs section

Contains parameters of requests to FrostFS.

The set_copies_number value can be overridden with X-Amz-Meta-Frostfs-Copies-Number (value is comma separated numbers: 1,2,3) header for PutObject, CopyObject, CreateMultipartUpload.

frostfs:
  set_copies_number: [0]
  client_cut: false
  buffer_max_size_for_put: 1048576 # 1mb
  tree_pool_max_attempts: 0
  graceful_close_on_switch_timeout: 10s

Parameter	Type	SIGHUP reload	Default value	Description
set_copies_number	[]uint32	yes	[0]	Numbers of the object copies (for each replica) to consider PUT to FrostFS successful. Default value [0] or empty list means that object will be processed according to the container's placement policy
client_cut	bool	yes	false	This flag enables client side object preparing.
buffer_max_size_for_put	uint64	yes	1048576	Sets max buffer size for read payload in put operations.
tree_pool_max_attempts	uint32	no	0	Sets max attempt to make successful tree request. Value 0 means the number of attempts equals to number of nodes in pool.
graceful_close_on_switch_timeout	duration	no	10s	Specifies the timeout after which unhealthy client be closed during rebalancing if it will become healthy back.

resolve_bucket section

Bucket name resolving parameters from and to container ID.

resolve_bucket:
  namespace_header: X-Frostfs-Namespace
  allow:
    - container
  deny:

Parameter	Type	SIGHUP reload	Default value	Description
namespace_header	string	yes	X-Frostfs-Namespace	Header to determine zone to resolve bucket name.
allow	[]string	no		List of container zones which are available to resolve. Mutual exclusive with deny list. Prioritized over deny list.
deny	[]string	no		List of container zones which are restricted to resolve. Mutual exclusive with allow list.

kludge section

Workarounds for non-standard use cases.

kludge:
  use_default_xmlns: false
  bypass_content_encoding_check_in_chunks: false
  default_namespaces: [ "", "root" ]

Parameter	Type	SIGHUP reload	Default value	Description
use_default_xmlns	bool	yes	false	Enable using default xml namespace http://s3.amazonaws.com/doc/2006-03-01/ when parse xml bodies.
bypass_content_encoding_check_in_chunks	bool	yes	false	Use this flag to be able to use chunked upload approach without having aws-chunked value in Content-Encoding header.
default_namespaces	[]string	yes	["","root"]	Namespaces that should be handled as default.

runtime section

Contains runtime parameters.

runtime:
  soft_memory_limit: 1gb

Parameter	Type	SIGHUP reload	Default value	Description
soft_memory_limit	size	yes	maxint64	Soft memory limit for the runtime. Zero or no value stands for no limit. If GOMEMLIMIT environment variable is set, the value from the configuration file will be ignored.

features section

Contains parameters for enabling features.

features:
   policy:
      deny_by_default: false
   md5:
      enabled: false

Parameter	Type	SIGHUP reload	Default value	Description
md5.enabled	bool	yes	false	Flag to enable return MD5 checksum in ETag headers and fields.
policy.deny_by_default	bool	yes	false	Enable denying access for request that doesn't match any policy chain rules.

web section

Contains web server configuration parameters.

web:
  read_timeout: 0
  read_header_timeout: 30s
  write_timeout: 0
  idle_timeout: 30s

Parameter	Type	SIGHUP reload	Default value	Description
read_timeout	duration	no	0	The maximum duration for reading the entire request, including the body. A zero or negative value means there will be no timeout.
read_header_timeout	duration	no	30s	The amount of time allowed to read request headers. If read_header_timeout is zero, the value of read_timeout is used. If both are zero, there is no timeout.
write_timeout	duration	no	0	The maximum duration before timing out writes of the response. A zero or negative value means there will be no timeout.
idle_timeout	duration	no	30s	The maximum amount of time to wait for the next request when keep-alives are enabled. If idle_timeout is zero, the value of read_timeout is used. If both are zero, there is no timeout.

frostfsid section

FrostfsID contract configuration. To enable this functionality the rpc_endpoint param must be also set.

frostfsid:
  contract: frostfsid.frostfs
  validation:
   enabled: false

Parameter	Type	SIGHUP reload	Default value	Description
contract	string	no	frostfsid.frostfs	FrostfsID contract hash (LE) or name in NNS.
validation.enabled	bool	no	false	Enables a check to only allow requests to users registered in the FrostfsID contract.

policy section

Policy contract configuration. To enable this functionality the rpc_endpoint param must be also set.

policy:
  contract: policy.frostfs

Parameter	Type	SIGHUP reload	Default value	Description
contract	string	no	policy.frostfs	Policy contract hash (LE) or name in NNS.

proxy section

Proxy contract configuration. To enable this functionality the rpc_endpoint param must be also set.

proxy:
  contract: proxy.frostfs

Parameter	Type	SIGHUP reload	Default value	Description
contract	string	no	proxy.frostfs	Proxy contract hash (LE) or name in NNS.

namespaces section

Namespaces configuration.

namespaces:
  config: namespace.json

Parameter	Type	SIGHUP reload	Default value	Description
config	string	yes		Path to json file with config value for namespaces.

namespaces.config subsection

Example of namespaces.json. Note that config values from namespaces.json can override config values for default namespaces (value for which are fetched from regular config value e.g. placement-policy). To override config values for default namespaces use namespace names that are provided in kludge.default_namespaces.

{
  "namespaces": {
    "namespace1": {
      "location_constraints": {
        "default": "REP 3",
        "test": "{\"replicas\":[{\"count\":1,\"selector\":\"\"}],\"containerBackupFactor\":0,\"selectors\":[],\"filters\":[],\"unique\":false}"
      },
      "copies_numbers": {
        "default": [ 0 ],
        "test": [ 1 ]
      }
    }
  }
}

retry section

Retry strategy configuration.

retry:
  max_attempts: 4
  max_backoff: 30s
  strategy: exponential

Parameter	Type	SIGHUP reload	Default value	Description
max_attemps	int	yes	4	Max amount of request attempts. Currently only for updating bucket settings request.
max_backoff	duration	yes	30s	Max delay before next attempt.
strategy	string	yes	exponential	Backoff strategy. exponential and constant are allowed.

containers section

Section for well-known containers to store s3-related data and settings.

containers:
  cors: AZjLTXfK4vs4ovxMic2xEJKSymMNLqdwq9JT64ASFCRj
  lifecycle: AZjLTXfK4vs4ovxMic2xEJKSymMNLqdwq9JT64ASFCRj

Parameter	Type	SIGHUP reload	Default value	Description
cors	string	no		Container name for CORS configurations. If not set, container of the bucket is used.
lifecycle	string	no		Container name for lifecycle configurations. If not set, container of the bucket is used.

vhs section

Configuration of virtual hosted addressing style.

vhs:
   enabled: false
   vhs_header: X-Frostfs-S3-VHS
   servername_header: X-Frostfs-Servername
   namespaces:
      "ns1": false
      "ns2": true

Parameter	Type	SIGHUP reload	Default value	Description
enabled	bool	yes	false	Enables the use of virtual host addressing for buckets at the application level.
vhs_header	string	yes	X-Frostfs-S3-VHS	Header for determining whether VHS is enabled for the request.
servername_header	string	yes	X-Frostfs-Servername	Header for determining servername.
namespaces	map[string]bool	yes		A map in which the keys are the name of the namespace, and the values are the flag responsible for enabling VHS for the specified namespace. Overrides global 'enabled' setting even when it is disabled.