frostfs-node/docs/storage-node-configuration.md
Dmitrii Stepanov 8180a0664f
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 6m1s
Build / Build Components (1.21) (pull_request) Successful in 7m37s
Build / Build Components (1.20) (pull_request) Successful in 7m52s
Tests and linters / Staticcheck (pull_request) Successful in 8m56s
Tests and linters / Lint (pull_request) Successful in 9m26s
Tests and linters / Tests (1.21) (pull_request) Successful in 15m5s
Tests and linters / Tests with -race (pull_request) Successful in 15m7s
DCO action / DCO (pull_request) Successful in 1m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m1s
[#887] node: Drop badger writecache implementation
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-22 13:00:54 +03:00

34 KiB

FrostFS Storage node configuration file

This section contains detailed FrostFS Storage node configuration file description including default config values and some tips to set up configurable values.

There are some custom types used for brevity:

  1. duration -- string consisting of a number and a suffix. Suffix examples include s (seconds), m (minutes), ms (milliseconds).
  2. size -- string consisting of a number and a suffix. Suffix examples include b (bytes, default), k (kibibytes), m (mebibytes), g (gibibytes).
  3. file mode -- octal number. Usually, it starts with 0 and contain 3 digits, corresponding to file access permissions for user, group and others.
  4. public key -- hex-encoded public key
  5. hash160 -- hex-encoded 20-byte hash of a deployed contract.

Structure

Section Description
logger Logging parameters
pprof PProf configuration
prometheus Prometheus metrics configuration
control Control service configuration
contracts Override FrostFS contracts hashes
morph N3 blockchain client configuration
apiclient FrostFS API client configuration
policer Policer service configuration
replicator Replicator service configuration
storage Storage engine configuration
runtime Runtime configuration

control section

control:
  authorized_keys:
    - 035839e45d472a3b7769a2a1bd7d54c4ccd4943c3b40f547870e83a8fcbfb3ce11
    - 028f42cfcb74499d7b15b35d9bff260a1c8d27de4f446a627406a382d8961486d6
  grpc:
    endpoint: 127.0.0.1:8090
Parameter Type Default value Description
authorized_keys []public key empty List of public keys which are used to authorize requests to the control service.
grpc.endpoint string empty Address that control service listener binds to.

grpc section

grpc:
  - endpoint: localhost:8080
    tls:
      enabled: true
      certificate: /path/to/cert.pem
      key: /path/to/key.pem
  - endpoint: internal.ip:8080
  - endpoint: external.ip:8080
    tls:
      enabled: true
      use_insecure_crypto: true

Contains an array of gRPC endpoint configurations. The following table describes the format of each element.

Parameter Type Default value Description
endpoint []string empty Address that service listener binds to.
tls TLS config Address that control service listener binds to.

tls subsection

Parameter Type Default value Description
enabled bool false Address that control service listener binds to.
certificate string Path to the TLS certificate.
key string Path to the key.
use_insecure_crypto bool false If true, ciphers considered insecure by Go stdlib are allowed to be used.

pprof section

Contains configuration for the pprof profiler.

Parameter Type Default value Description
enabled bool false Flag to enable the service.
address string Address that service listener binds to.
shutdown_timeout duration 30s Time to wait for a graceful shutdown.
debug Debug config Optional profiles configuration

debug subsection

Contains optional profiles configuration.

Parameter Type Default value Description
block_rate int 0 Controls the block profiler. Non-positive values disable profiler reports. For more information: https://pkg.go.dev/runtime@go1.20.3#SetBlockProfileRate.
mutex_rate int 0 Controls the mutex profiler. Non-positive values disable profiler reports. For more information: https://pkg.go.dev/runtime@go1.20.3#SetMutexProfileFraction.

prometheus section

Contains configuration for the prometheus metrics service.

Parameter Type Default value Description
enabled bool false Flag to enable the service.
address string Address that service listener binds to.
shutdown_timeout duration 30s Time to wait for a graceful shutdown.

logger section

Contains logger parameters.

logger:
  level: info
Parameter Type Default value Description
level string info Logging level.
Possible values: debug, info, warn, error, dpanic, panic, fatal

contracts section

Contains override values for FrostFS side-chain contract hashes. Most of the time contract hashes are fetched from the NNS contract, so this section can be omitted.

contracts:
  balance: 5263abba1abedbf79bb57f3e40b50b4425d2d6cd
  container: 5d084790d7aa36cea7b53fe897380dab11d2cd3c
  netmap: 0cce9e948dca43a6b592efe59ddb4ecb89bdd9ca
  proxy: ad7c6b55b737b696e5c82c85445040964a03e97f
Parameter Type Default value Description
balance hash160 Balance contract hash.
container hash160 Container contract hash.
netmap hash160 Netmap contract hash.

morph section

morph:
  dial_timeout: 30s
  cache_ttl: 15s
  rpc_endpoint:
    - address: wss://rpc1.morph.frostfs.info:40341/ws
      priority: 1
    - address: wss://rpc2.morph.frostfs.info:40341/ws
      priority: 2
  switch_interval: 2m
Parameter Type Default value Description
dial_timeout duration 5s Timeout for dialing connections to N3 RPCs.
cache_ttl duration Morph block time Sidechain cache TTL value (min interval between similar calls).
Negative value disables caching.
Cached entities: containers, container lists, eACL tables.
rpc_endpoint list of endpoint descriptions Array of endpoint descriptions.
switch_interval duration 2m Time interval between the attempts to connect to the highest priority RPC node if the connection is not established yet.

rpc_endpoint subsection

Parameter Type Default value Description
address string WebSocket N3 endpoint.
priority int 1 Priority of an endpoint. Endpoint with a higher priority (lower configuration value) has more chance of being used. Endpoints with equal priority are iterated over randomly; a negative priority is interpreted as 1.

storage section

Local storage engine configuration.

Parameter Type Default value Description
shard_pool_size int 20 Pool size for shard workers. Limits the amount of concurrent PUT operations on each shard.
shard_ro_error_threshold int 0 Maximum amount of storage errors to encounter before shard automatically moves to Degraded or ReadOnly mode.
low_mem bool false Reduce memory consumption by reducing performance.
shard Shard config Configuration for separate shards.

shard subsection

Contains configuration for each shard. Keys must be consecutive numbers starting from zero. default subsection has the same format and specifies defaults for missing values. The following table describes configuration for each shard.

Parameter Type Default value Description
compress bool false Flag to enable compression.
compression_exclude_content_types []string List of content-types to disable compression for. Content-type is taken from Content-Type object attribute. Each element can contain a star * as a first (last) character, which matches any prefix (suffix).
compression_estimate_compressibility bool false If true, then noramalized compressibility estimation is used to decide compress data or not.
compression_estimate_compressibility_threshold float 0.1 Normilized compressibility estimate threshold: data will compress if estimation if greater than this value.
mode string read-write Shard Mode.
Possible values: read-write, read-only, degraded, degraded-read-only, disabled
resync_metabase bool false Flag to enable metabase resync on start.
writecache Writecache config Write-cache configuration.
metabase Metabase config Metabase configuration.
blobstor Blobstor config Blobstor configuration.
small_object_size size 1M Maximum size of an object stored in blobovnicza tree.
gc GC config GC configuration.

blobstor subsection

Contains a list of substorages each with it's own type. Currently only 2 types are supported: fstree and blobovnicza.

blobstor:
  - type: blobovnicza
    path: /path/to/blobstor
    depth: 1
    width: 4
  - type: fstree
    path: /path/to/blobstor/blobovnicza
    perm: 0644
    size: 4194304
    depth: 1
    width: 4
    opened_cache_capacity: 50
    opened_cache_ttl: 5m

Common options for sub-storages

Parameter Type Default value Description
path string Path to the root of the blobstor.
perm file mode 0660 Default permission for created files and directories.

fstree type options

Parameter Type Default value Description
path string Path to the root of the blobstor.
perm file mode 0660 Default permission for created files and directories.
depth int 4 File-system tree depth.

blobovnicza type options

Parameter Type Default value Description
path string Path to the root of the blobstor.
perm file mode 0660 Default permission for created files and directories.
size size 1 G Maximum size of a single blobovnicza
depth int 2 Blobovnicza tree depth.
width int 16 Blobovnicza tree width.
opened_cache_capacity int 16 Maximum number of simultaneously opened blobovniczas.
opened_cache_ttl duration 0 TTL in cache for opened blobovniczas(disabled by default). In case of heavy random-read and 10 shards each with 10_000 databases and accessing 400 objects per-second we will access each db approximately once per ((10 * 10_000 / 400) = 250 seconds <= 300 seconds = 5 min). Also take in mind that in this scenario they will probably be closed earlier because of the cache capacity, so bigger values are likely to be of no use.

gc subsection

Contains garbage-collection service configuration. It iterates over the blobstor and removes object the node no longer needs.

gc:
  remover_batch_size: 200
  remover_sleep_interval: 5m
  expired_collector_batch_size: 500
  expired_collector_worker_count: 5
Parameter Type Default value Description
remover_batch_size int 100 Amount of objects to grab in a single batch.
remover_sleep_interval duration 1m Time to sleep between iterations.
expired_collector_batch_size int 500 Max amount of expired objects to grab in a single batch.
expired_collector_worker_count int 5 Max amount of concurrent expired objects workers.

metabase subsection

metabase:
  path: /path/to/meta.db
  perm: 0644
  max_batch_size: 200
  max_batch_delay: 20ms
Parameter Type Default value Description
path string Path to the metabase file.
perm file mode 0660 Permissions to set for the database file.
max_batch_size int 1000 Maximum amount of write operations to perform in a single transaction.
max_batch_delay duration 10ms Maximum delay before a batch starts.

writecache subsection

writecache:
  enabled: true
  path: /path/to/writecache
  capacity: 4294967296
  small_object_size: 16384
  max_object_size: 134217728
  flush_worker_count: 30
Parameter Type Default value Description
path string Path to the metabase file.
capacity size unrestricted Approximate maximum size of the writecache. If the writecache is full, objects are written to the blobstor directly.
small_object_size size 32K Maximum object size for "small" objects. This objects are stored in a key-value database instead of a file-system.
max_object_size size 64M Maximum object size allowed to be stored in the writecache.
flush_worker_count int 20 Amount of background workers that move data from the writecache to the blobstor.
max_batch_size int 1000 Maximum amount of small object PUT operations to perform in a single transaction.
max_batch_delay duration 10ms Maximum delay before a batch starts.

node section

node:
  wallet:
    path: /path/to/wallet.json
    address: NcpJzXcSDrh5CCizf4K9Ro6w4t59J5LKzz
    password: password
  addresses:
    - grpc://external.ip:8082
  attribute:
    - "Price:11"
    - "UN-LOCODE:RU MSK"
    - "key:value"
  relay: false
  persistent_sessions:
    path: /sessions
  persistent_state:
    path: /state
  notification:
    enabled: true
    endpoint: tls://localhost:4222
    timeout: 6s
    default_topic: topic
    certificate: /path/to/cert.pem
    key: /path/to/key.pem
    ca: /path/to/ca.pem
Parameter Type Default value Description
key string Path to the binary-encoded private key.
wallet Wallet config Wallet configuration. Has no effect if key is provided.
addresses []string Addresses advertised in the netmap.
attribute []string Node attributes as a list of key-value pairs in <key>:<value> format.
relay bool Enable relay mode.
persistent_sessions Persistent sessions config Persistent session token store configuration.
persistent_state Persistent state config Persistent state configuration.
notification Notification config NATS configuration.

wallet subsection

N3 wallet configuration.

Parameter Type Default value Description
path string Path to the wallet file.
address string Wallet address to use.
password string Password to open the wallet.

persistent_sessions subsection

Contains persistent session token store configuration. By default sessions do not persist between restarts.

Parameter Type Default value Description
path string Path to the database.

persistent_state subsection

Configures persistent storage for auxiliary information, such as last seen block height. It is used to correctly handle node restarts or crashes.

Parameter Type Default value Description
path string .frostfs-storage-state Path to the database.

notification subsection

This is an advanced section, use with caution.

Parameter Type Default value Description
enabled bool false Flag to enable the service.
endpoint string NATS endpoint to connect to.
timeout duration 5s Timeout for the object notification operation.
default_topic string node's public key Default topic to use if an object has no corresponding attribute.
certificate string Path to the client certificate.
key string Path to the client key.
ca string Override root CA used to verify server certificates.

apiclient section

Configuration for the FrostFS API client used for communication with other FrostFS nodes.

apiclient:
  dial_timeout: 15s
  stream_timeout: 20s
  reconnect_timeout: 30s
Parameter Type Default value Description
dial_timeout duration 5s Timeout for dialing connections to other storage or inner ring nodes.
stream_timeout duration 15s Timeout for individual operations in a streaming RPC.
reconnect_timeout duration 30s Time to wait before reconnecting to a failed node.

policer section

Configuration for the Policer service. It ensures that object is stored according to the intended policy.

policer:
  head_timeout: 15s
Parameter Type Default value Description
head_timeout duration 5s Timeout for performing the HEAD operation.

replicator section

Configuration for the Replicator service.

replicator:
  put_timeout: 15s
  pool_size: 10
Parameter Type Default value Description
put_timeout duration 5s Timeout for performing the PUT operation.
pool_size int Equal to object.put.remote_pool_size Maximum amount of concurrent replications.

object section

Contains object-service related parameters.

object:
  put:
    remote_pool_size: 100
Parameter Type Default value Description
delete.tombstone_lifetime int 5 Tombstone lifetime for removed objects in epochs.
put.remote_pool_size int 10 Max pool size for performing remote PUT operations. Used by Policer and Replicator services.
put.local_pool_size int 10 Max pool size for performing local PUT operations. Used by Policer and Replicator services.

runtime section

Contains runtime parameters.

runtime:
  soft_memory_limit: 1GB
Parameter Type Default value Description
soft_memory_limit size 0 Soft memory limit for the runtime. Zero or no value stands for no limit. If GOMEMLIMIT environment variable is set, the value from the configuration file will be ignored.