frostfs-node/docs/storage-node-configuration.md

FrostFS Storage node configuration file

This section contains detailed FrostFS Storage node configuration file description including default config values and some tips to set up configurable values.

There are some custom types used for brevity:

1. duration -- string consisting of a number and a suffix. Suffix examples include s (seconds), m (minutes), ms (milliseconds).
2. size -- string consisting of a number and a suffix. Suffix examples include b (bytes, default), k (kibibytes), m (mebibytes), g (gibibytes).
3. file mode -- octal number. Usually, it starts with 0 and contain 3 digits, corresponding to file access permissions for user, group and others.
4. public key -- hex-encoded public key
5. hash160 -- hex-encoded 20-byte hash of a deployed contract.

Structure

Section	Description
logger	Logging parameters
pprof	PProf configuration
prometheus	Prometheus metrics configuration
control	Control service configuration
contracts	Override FrostFS contracts hashes
morph	N3 blockchain client configuration
apiclient	FrostFS API client configuration
policer	Policer service configuration
replicator	Replicator service configuration
storage	Storage engine configuration
runtime	Runtime configuration
audit	Audit configuration
multinet	Multinet configuration

control section

control:
  authorized_keys:
    - 035839e45d472a3b7769a2a1bd7d54c4ccd4943c3b40f547870e83a8fcbfb3ce11
    - 028f42cfcb74499d7b15b35d9bff260a1c8d27de4f446a627406a382d8961486d6
  grpc:
    endpoint: 127.0.0.1:8090

Parameter	Type	Default value	Description
authorized_keys	[]public key	empty	List of public keys which are used to authorize requests to the control service.
grpc.endpoint	string	empty	Address that control service listener binds to.

grpc section

grpc:
  - endpoint: localhost:8080
    tls:
      enabled: true
      certificate: /path/to/cert.pem
      key: /path/to/key.pem
  - endpoint: internal.ip:8080
  - endpoint: external.ip:8080
    tls:
      enabled: true
      use_insecure_crypto: true

Contains an array of gRPC endpoint configurations. The following table describes the format of each element.

Parameter	Type	Default value	Description
endpoint	[]string	empty	Address that service listener binds to.
tls	TLS config		Address that control service listener binds to.

tls subsection

Parameter	Type	Default value	Description
enabled	bool	false	Address that control service listener binds to.
certificate	string		Path to the TLS certificate.
key	string		Path to the key.
use_insecure_crypto	bool	false	If true, ciphers considered insecure by Go stdlib are allowed to be used.

pprof section

Contains configuration for the pprof profiler.

Parameter	Type	Default value	Description
enabled	bool	false	Flag to enable the service.
address	string		Address that service listener binds to.
shutdown_timeout	duration	30s	Time to wait for a graceful shutdown.
debug	Debug config		Optional profiles configuration

debug subsection

Contains optional profiles configuration.

Parameter	Type	Default value	Description
block_rate	int	0	Controls the block profiler. Non-positive values disable profiler reports. For more information: https://pkg.go.dev/runtime@go1.20.3#SetBlockProfileRate.
mutex_rate	int	0	Controls the mutex profiler. Non-positive values disable profiler reports. For more information: https://pkg.go.dev/runtime@go1.20.3#SetMutexProfileFraction.

prometheus section

Contains configuration for the prometheus metrics service.

Parameter	Type	Default value	Description
enabled	bool	false	Flag to enable the service.
address	string		Address that service listener binds to.
shutdown_timeout	duration	30s	Time to wait for a graceful shutdown.

logger section

Contains logger parameters.

logger:
  level: info

Parameter	Type	Default value	Description
level	string	info	Logging level. Possible values: debug, info, warn, error, dpanic, panic, fatal

contracts section

Contains override values for FrostFS side-chain contract hashes. Most of the time contract hashes are fetched from the NNS contract, so this section can be omitted.

contracts:
  balance: 5263abba1abedbf79bb57f3e40b50b4425d2d6cd
  container: 5d084790d7aa36cea7b53fe897380dab11d2cd3c
  netmap: 0cce9e948dca43a6b592efe59ddb4ecb89bdd9ca
  proxy: ad7c6b55b737b696e5c82c85445040964a03e97f

Parameter	Type	Default value	Description
balance	hash160		Balance contract hash.
container	hash160		Container contract hash.
netmap	hash160		Netmap contract hash.

morph section

morph:
  dial_timeout: 30s
  cache_ttl: 15s
  ape_chain_cache_size: 10000
  rpc_endpoint:
    - address: wss://rpc1.morph.frostfs.info:40341/ws
      priority: 1
    - address: wss://rpc2.morph.frostfs.info:40341/ws
      priority: 2
  switch_interval: 2m

Parameter	Type	Default value	Description
dial_timeout	duration	5s	Timeout for dialing connections to N3 RPCs.
cache_ttl	duration	Morph block time	Sidechain cache TTL value (min interval between similar calls). Negative value disables caching. Cached entities: containers, container lists, eACL tables.
rpc_endpoint	list of endpoint descriptions		Array of endpoint descriptions.
switch_interval	duration	2m	Time interval between the attempts to connect to the highest priority RPC node if the connection is not established yet.
ape_chain_cache_size	int	10000	Size of the morph cache for APE chains.

rpc_endpoint subsection

Parameter	Type	Default value	Description
address	string		WebSocket N3 endpoint.
priority	int	1	Priority of an endpoint. Endpoint with a higher priority (lower configuration value) has more chance of being used. Endpoints with equal priority are iterated over randomly; a negative priority is interpreted as 1.

storage section

Local storage engine configuration.

Parameter	Type	Default value	Description
shard_pool_size	int	20	Pool size for shard workers. Limits the amount of concurrent PUT operations on each shard.
shard_ro_error_threshold	int	0	Maximum amount of storage errors to encounter before shard automatically moves to Degraded or ReadOnly mode.
low_mem	bool	false	Reduce memory consumption by reducing performance.
shard	Shard config		Configuration for separate shards.

shard subsection

Contains configuration for each shard. Keys must be consecutive numbers starting from zero. default subsection has the same format and specifies defaults for missing values. The following table describes configuration for each shard.

Parameter	Type	Default value	Description
compress	bool	false	Flag to enable compression.
compression_exclude_content_types	[]string		List of content-types to disable compression for. Content-type is taken from Content-Type object attribute. Each element can contain a star * as a first (last) character, which matches any prefix (suffix).
compression_estimate_compressibility	bool	false	If true, then noramalized compressibility estimation is used to decide compress data or not.
compression_estimate_compressibility_threshold	float	0.1	Normilized compressibility estimate threshold: data will compress if estimation if greater than this value.
mode	string	read-write	Shard Mode. Possible values: read-write, read-only, degraded, degraded-read-only, disabled
resync_metabase	bool	false	Flag to enable metabase resync on start.
resync_metabase_worker_count	int	1000	Count of concurrent workers to resync metabase.
writecache	Writecache config		Write-cache configuration.
metabase	Metabase config		Metabase configuration.
blobstor	Blobstor config		Blobstor configuration.
small_object_size	size	1M	Maximum size of an object stored in blobovnicza tree.
gc	GC config		GC configuration.

blobstor subsection

Contains a list of substorages each with it's own type. Currently only 2 types are supported: fstree and blobovnicza.

blobstor:
  - type: blobovnicza
    path: /path/to/blobstor
    depth: 1
    width: 4
  - type: fstree
    path: /path/to/blobstor/blobovnicza
    perm: 0644
    size: 4194304
    depth: 1
    width: 4
    opened_cache_capacity: 50
    opened_cache_ttl: 5m
    opened_cache_exp_interval: 15s

Common options for sub-storages

Parameter	Type	Default value	Description
path	string		Path to the root of the blobstor.
perm	file mode	0660	Default permission for created files and directories.

fstree type options

Parameter	Type	Default value	Description
path	string		Path to the root of the blobstor.
perm	file mode	0660	Default permission for created files and directories.
depth	int	4	File-system tree depth.

blobovnicza type options

Parameter	Type	Default value	Description
path	string		Path to the root of the blobstor.
perm	file mode	0660	Default permission for created files and directories.
size	size	1 G	Maximum size of a single blobovnicza
depth	int	2	Blobovnicza tree depth.
width	int	16	Blobovnicza tree width.
opened_cache_capacity	int	16	Maximum number of simultaneously opened blobovniczas.
opened_cache_ttl	duration	0	TTL in cache for opened blobovniczas(disabled by default). In case of heavy random-read and 10 shards each with 10_000 databases and accessing 400 objects per-second we will access each db approximately once per ((10 * 10_000 / 400) = 250 seconds <= 300 seconds = 5 min). Also take in mind that in this scenario they will probably be closed earlier because of the cache capacity, so bigger values are likely to be of no use.
opened_cache_exp_interval	duration	15s	Cache cleanup interval for expired blobovnicza's.
init_worker_count	int	5	Maximum number of concurrent initialization workers.
rebuild_drop_timeout	duration	10s	Timeout before drop empty blobovnicza file during rebuild.

gc subsection

Contains garbage-collection service configuration. It iterates over the blobstor and removes object the node no longer needs.

gc:
  remover_batch_size: 200
  remover_sleep_interval: 5m
  expired_collector_batch_size: 500
  expired_collector_worker_count: 5

Parameter	Type	Default value	Description
remover_batch_size	int	100	Amount of objects to grab in a single batch.
remover_sleep_interval	duration	1m	Time to sleep between iterations.
expired_collector_batch_size	int	500	Max amount of expired objects to grab in a single batch.
expired_collector_worker_count	int	5	Max amount of concurrent expired objects workers.

metabase subsection

metabase:
  path: /path/to/meta.db
  perm: 0644
  max_batch_size: 200
  max_batch_delay: 20ms

Parameter	Type	Default value	Description
path	string		Path to the metabase file.
perm	file mode	0660	Permissions to set for the database file.
max_batch_size	int	1000	Maximum amount of write operations to perform in a single transaction.
max_batch_delay	duration	10ms	Maximum delay before a batch starts.

writecache subsection

writecache:
  enabled: true
  path: /path/to/writecache
  capacity: 4294967296
  max_object_size: 134217728
  flush_worker_count: 30

Parameter	Type	Default value	Description
path	string		Path to the metabase file.
capacity	size	1G	Approximate maximum size of the writecache. If the writecache is full, objects are written to the blobstor directly.
max_object_count	int	unrestricted	Approximate maximum objects count in the writecache. If the writecache is full, objects are written to the blobstor directly.
max_object_size	size	64M	Maximum object size allowed to be stored in the writecache.
flush_worker_count	int	20	Amount of background workers that move data from the writecache to the blobstor.
max_flushing_objects_size	size	512M	Max total size of background flushing objects.

node section

node:
  wallet:
    path: /path/to/wallet.json
    address: NcpJzXcSDrh5CCizf4K9Ro6w4t59J5LKzz
    password: password
  addresses:
    - grpc://external.ip:8082
  attribute:
    - "Price:11"
    - "UN-LOCODE:RU MSK"
    - "key:value"
  relay: false
  persistent_sessions:
    path: /sessions
  persistent_state:
    path: /state

Parameter	Type	Default value	Description
key	string		Path to the binary-encoded private key.
wallet	Wallet config		Wallet configuration. Has no effect if key is provided.
addresses	[]string		Addresses advertised in the netmap.
attribute	[]string		Node attributes as a list of key-value pairs in <key>:<value> format.
relay	bool		Enable relay mode.
persistent_sessions	Persistent sessions config		Persistent session token store configuration.
persistent_state	Persistent state config		Persistent state configuration.

wallet subsection

N3 wallet configuration.

Parameter	Type	Default value	Description
path	string		Path to the wallet file.
address	string		Wallet address to use.
password	string		Password to open the wallet.

persistent_sessions subsection

Contains persistent session token store configuration. By default sessions do not persist between restarts.

Parameter	Type	Default value	Description
path	string		Path to the database.

persistent_state subsection

Configures persistent storage for auxiliary information, such as last seen block height. It is used to correctly handle node restarts or crashes.

Parameter	Type	Default value	Description
path	string	.frostfs-storage-state	Path to the database.

apiclient section

Configuration for the FrostFS API client used for communication with other FrostFS nodes.

apiclient:
  dial_timeout: 15s
  stream_timeout: 20s
  reconnect_timeout: 30s

Parameter	Type	Default value	Description
dial_timeout	duration	5s	Timeout for dialing connections to other storage or inner ring nodes.
stream_timeout	duration	15s	Timeout for individual operations in a streaming RPC.
reconnect_timeout	duration	30s	Time to wait before reconnecting to a failed node.

policer section

Configuration for the Policer service. It ensures that object is stored according to the intended policy.

policer:
  head_timeout: 15s

Parameter	Type	Default value	Description
head_timeout	duration	5s	Timeout for performing the HEAD operation.

replicator section

Configuration for the Replicator service.

replicator:
  put_timeout: 15s
  pool_size: 10

Parameter	Type	Default value	Description
put_timeout	duration	5s	Timeout for performing the PUT operation.
pool_size	int	Equal to object.put.remote_pool_size	Maximum amount of concurrent replications.

object section

Contains object-service related parameters.

object:
  put:
    remote_pool_size: 100
  get:
    priority:
      - $attribute:ClusterName

Parameter	Type	Default value	Description
delete.tombstone_lifetime	int	5	Tombstone lifetime for removed objects in epochs.
put.remote_pool_size	int	10	Max pool size for performing remote PUT operations. Used by Policer and Replicator services.
put.local_pool_size	int	10	Max pool size for performing local PUT operations. Used by Policer and Replicator services.
get.priority	[]string		List of metrics of nodes for prioritization. Used for computing response on GET requests.

runtime section

Contains runtime parameters.

runtime:
  soft_memory_limit: 1GB

Parameter	Type	Default value	Description
soft_memory_limit	size	0	Soft memory limit for the runtime. Zero or no value stands for no limit. If GOMEMLIMIT environment variable is set, the value from the configuration file will be ignored.

audit section

Contains audit parameters.

audit:
  enabled: true

Parameter	Type	Default value	Description
enabled	bool	false	If true then audit event logs will be recorded.

multinet section

Contains multinet parameters.

multinet:
  enabled: true
  subnets:
    - mask: 192.168.219.174/24
      source_ips:
        - 192.168.218.185
        - 192.168.219.185
    - mask: 10.78.70.74/24
      source_ips:
        - 10.78.70.185
        - 10.78.71.185
  balancer: roundrobin
  restrict: false
  fallback_delay: 350ms

Parameter	Type	Default value	Description
enabled	bool	false	If true then source-based routing is enabled.
subnets	subnet	empty	Resulting subnets.
balancer	string	""	Balancer to select network interfaces, allowed values are "" (no balancing, use first suitable interface) or "roundrobin".
restrict	bool	false	If true then any requests that do not match subnets will fail.
fallback_delay	duration	350ms	Delay before fallback to secondary IP addresses in case of hostname resolve.