From 5e1636e50cfc141c26d63488edf3f928e5b1ddf7 Mon Sep 17 00:00:00 2001 From: Evgenii Stratonikov Date: Thu, 14 Jul 2022 13:39:35 +0300 Subject: [PATCH] [#1604] docs: Describe storage node configuration Signed-off-by: Evgenii Stratonikov --- docs/storage-node-configuration.md | 405 +++++++++++++++++++++++++++++ 1 file changed, 405 insertions(+) create mode 100644 docs/storage-node-configuration.md diff --git a/docs/storage-node-configuration.md b/docs/storage-node-configuration.md new file mode 100644 index 00000000..aec70acb --- /dev/null +++ b/docs/storage-node-configuration.md @@ -0,0 +1,405 @@ +# NeoFS Storage node configuration file + +This section contains detailed NeoFS Storage node configuration file description +including default config values and some tips to set up configurable values. + +There are some custom types used for brevity: +1. `duration` -- string consisting of a number and a suffix. Suffix examples include `s` (seconds), `m` (minutes), `ms` (milliseconds). +2. `size` -- string consisting of a number and a suffix. Suffix examples include `b` (bytes, default), `k` (kibibytes), `m` (mebibytes), `g` (gibibytes). +3. `file mode` -- octal number. Usually, it starts with `0` and contain 3 digits, corresponding to file access permissions for user, group and others. +4. `public key` -- hex-encoded public key +5. `hash160` -- hex-encoded 20-byte hash of a deployed contract. + +# Structure + +| Section | Description | +|--------------|---------------------------------------------------------| +| `logger` | [Logging parameters](#logger-section) | +| `pprof` | [PProf configuration](#pprof-section) | +| `prometheus` | [Prometheus metrics configuration](#prometheus-section) | +| `control` | [Control service configuration](#control-section) | +| `contracts` | [Override NeoFS contracts hashes](#contracts-section) | +| `morph` | [N3 blockchain client configuration](#morph-section) | +| `apiclient` | [NeoFS API client configuration](#apiclient-section) | +| `policer` | [Policer service configuration](#policer-section) | +| `replicator` | [Replicator service configuration](#replicator-section) | +| `storage` | [Storage engine configuration](#storage-section) | + + +# `control` section +```yaml +control: + authorized_keys: + - 035839e45d472a3b7769a2a1bd7d54c4ccd4943c3b40f547870e83a8fcbfb3ce11 + - 028f42cfcb74499d7b15b35d9bff260a1c8d27de4f446a627406a382d8961486d6 + grpc: + endpoint: 127.0.0.1:8090 +``` +| Parameter | Type | Default value | Description | +|-------------------|----------------|---------------|----------------------------------------------------------------------------------| +| `authorized_keys` | `[]public key` | empty | List of public keys which are used to authorize requests to the control service. | +| `grpc.endpoint` | `string` | empty | Address that control service listener binds to. | + +# `grpc` section +```yaml +grpc: + - endpoint: localhost:8080 + tls: + enabled: true + certificate: /path/to/cert.pem + key: /path/to/key.pem + - endpoint: internal.ip:8080 + - endpoint: external.ip:8080 + tls: + enabled: true + use_insecure_crypto: true +``` +Contains an array of gRPC endpoint configurations. The following table describes the format of each +element. + +| Parameter | Type | Default value | Description | +|---------------------------|-------------------------------|---------------|---------------------------------------------------------------------------| +| `endpoint` | `[]string` | empty | Address that service listener binds to. | +| `tls` | [TLS config](#tls-subsection) | | Address that control service listener binds to. | + +## `tls` subsection + +| Parameter | Type | Default value | Description | +|-----------------------|----------|---------------|---------------------------------------------------------------------------| +| `enabled` | `bool` | `false` | Address that control service listener binds to. | +| `certificate` | `string` | | Path to the TLS certificate. | +| `key` | `string` | | Path to the key. | +| `use_insecure_crypto` | `bool` | `false` | If true, ciphers considered insecure by Go stdlib are allowed to be used. | + +# `pprof` section + +Contains configuration for the `pprof` profiler. + +| Parameter | Type | Default value | Description | +|--------------------|------------|---------------|-----------------------------------------| +| `enabled` | `bool` | `false` | Flag to enable the service. | +| `address` | `string` | | Address that service listener binds to. | +| `shutdown_timeout` | `duration` | `30s` | Time to wait for a graceful shutdown. | + + +# `prometheus` section + +Contains configuration for the `prometheus` metrics service. + +| Parameter | Type | Default value | Description | +|--------------------|------------|---------------|-----------------------------------------| +| `enabled` | `bool` | `false` | Flag to enable the service. | +| `address` | `string` | | Address that service listener binds to. | +| `shutdown_timeout` | `duration` | `30s` | Time to wait for a graceful shutdown. | + +# `logger` section +Contains logger parameters. + +```yaml +logger: + level: info +``` + +| Parameter | Type | Default value | Description | +|-----------|----------|---------------|---------------------------------------------------------------------------------------------------| +| `level` | `string` | `info` | Logging level.
Possible values: `debug`, `info`, `warn`, `error`, `dpanic`, `panic`, `fatal` | + +# `contracts` section +Contains override values for NeoFS side-chain contract hashes. Most of the time contract +hashes are fetched from the NNS contract, so this section can be omitted. + +```yaml +contracts: + balance: 5263abba1abedbf79bb57f3e40b50b4425d2d6cd + container: 5d084790d7aa36cea7b53fe897380dab11d2cd3c + netmap: 0cce9e948dca43a6b592efe59ddb4ecb89bdd9ca + reputation: 441995f631c1da2b133462b71859494a5cd45e90 + proxy: ad7c6b55b737b696e5c82c85445040964a03e97f +``` + +| Parameter | Type | Default value | Description | +|--------------|-----------|---------------|---------------------------| +| `audit` | `hash160` | | Audit contract hash. | +| `balance` | `hash160` | | Balance contract hash. | +| `container` | `hash160` | | Container contract hash. | +| `netmap` | `hash160` | | Netmap contract hash. | +| `reputation` | `hash160` | | Reputation contract hash. | +| `subnet` | `hash160` | | Subnet contract hash. | + +# `morph` section + +```yaml +morph: + dial_timeout: 30s + disable_cache: true + rpc_endpoint: + - wss://rpc1.morph.fs.neo.org:40341/ws + - wss://rpc2.morph.fs.neo.org:40341/ws + ``` + +| Parameter | Type | Default value | Description | +|-----------------|------------|---------------|---------------------------------------------------------------------------------------------------------------------------------| +| `dial_timeout` | `duration` | `5s` | Timeout for dialing connections to N3 RPCs. | +| `disable_cache` | `bool` | `false` | Flag to disable TTL cache for some side-chain operations.
NOTE: Setting this to `true` can slow down the node considerably. | +| `rpc_endpoint` | `[]string` | | Array of _websocket_ N3 endpoints. | + +# `storage` section + +Local storage engine configuration. + +| Parameter | Type | Default value | Description | +|----------------------------|-----------------------------------|---------------|------------------------------------------------------------------------------------------------------------------| +| `shard_pool_size` | `int` | `20` | Pool size for shard workers. Limits the amount of concurrent `PUT` operations on each shard. | +| `shard_ro_error_threshold` | `int` | `0` | Maximum amount of storage errors to encounter before shard automatically moves to `Degraded` or `ReadOnly` mode. | +| `shard` | [Shard config](#shard-subsection) | | Configuration for separate shards. | + +## `shard` subsection + +Contains configuration for each shard. Keys must be consecutive numbers starting from zero. +`default` subsection has the same format and specifies defaults for missing values. +The following table describes configuration for each shard. + +| Parameter | Type | Default value | Description | +|-------------------|---------------------------------------------|---------------|------------------------------------------| +| `resync_metabase` | `bool` | `false` | Flag to enable metabase resync on start. | +| `writecache` | [Writecache config](#writecache-subsection) | | Write-cache configuration. | +| `metabase` | [Metabase config](#metabase-subsection) | | Metabase configuration. | +| `blobstor` | [Blobstor config](#blobstor-subsection) | | Blobstor configuration. | +| `gc` | [GC config](#gc-subsection) | | GC configuration. | + +### `blobstor` subsection + +```yaml +blobstor: + path: /path/to/blobstor + perm: 0644 + compress: true + compression_exclude_content_types: + - audio/* + - video/* + depth: 5 + small_object_size: 102400 + blobovnicza: + size: 4194304 + depth: 1 + width: 4 + opened_cache_capacity: 50 +``` +| Parameter | Type | Default value | Description | +|-------------------------------------|-----------------------------------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `path` | `string` | | Path to the root of the blobstor. | +| `perm` | file mode | `0660` | Default permission for created files and directories. | +| `compress` | `bool` | `false` | Flag to enable compression. | +| `compression_exclude_content_types` | `[]string` | | List of content-types to disable compression for. Content-type is taken from `Content-Type` object attribute. Each element can contain a star `*` as a first (last) character, which matches any prefix (suffix). | +| `depth` | `int` | `4` | Depth of the file-system tree for large objects. Must be in range 1..31. | +| `small_object_size` | `size` | `1M` | Maximum size of an object stored in blobovnicza tree. | +| `blobovnicza` | [Blobovnicza config](#blobovnicza-subsection) | | Blobovnicza tree configuration. | + +#### `blobovnicza` subsection + +| Parameter | Type | Default value | Description | +|-------------------------|----------|---------------|-------------------------------------------------------| +| `path` | `string` | | Path to the root of the blobovnicza tree. | +| `size` | `size` | `1 G` | Maximum size of a single blobovnicza | +| `depth` | `int` | `2` | Blobovnicza tree depth. | +| `width` | `int` | `16` | Blobovnicza tree width. | +| `opened_cache_capacity` | `int` | `16` | Maximum number of simultaneously opened blobovniczas. | + +### `gc` subsection + +Contains garbage-collection service configuration. It iterates over the blobstor and removes object the node no longer needs. + +```yaml +gc: + remover_batch_size: 200 + remover_sleep_interval: 5m +``` + +| Parameter | Type | Default value | Description | +|--------------------------|------------|---------------|----------------------------------------------| +| `remover_batch_size` | `int` | `100` | Amount of objects to grab in a single batch. | +| `remover_sleep_interval` | `duration` | `1m` | Time to sleep between iterations. | + +### `metabase` subsection + +```yaml +metabase: + path: /path/to/meta.db + perm: 0644 + max_batch_size: 200 + max_batch_delay: 20ms +``` + +| Parameter | Type | Default value | Description | +|-------------------|------------|---------------|------------------------------------------------------------------------| +| `path` | `string` | | Path to the metabase file. | +| `perm` | file mode | `0660` | Permissions to set for the database file. | +| `max_batch_size` | `int` | `1000` | Maximum amount of write operations to perform in a single transaction. | +| `max_batch_delay` | `duration` | `10ms` | Maximum delay before a batch starts. | + +### `writecache` subsection + +```yaml +writecache: + enabled: true + path: /path/to/writecache + capacity: 4294967296 + memcache_capacity: 2147483648 + small_object_size: 16384 + max_object_size: 134217728 + workers_number: 30 +``` + +| Parameter | Type | Default value | Description | +|----------------------|------------|---------------|----------------------------------------------------------------------------------------------------------------------| +| `path` | `string` | | Path to the metabase file. | +| `capacity` | `size` | unrestricted | Approximate maximum size of the writecache. If the writecache is full, objects are written to the blobstor directly. | +| `memcache_capacity` | `size` | `1G` | Maximum size of the memory-cache. | +| `small_object_size` | `size` | `32K` | Maximum object size for "small" objects. This objects are stored in a key-value database instead of a file-system. | +| `max_object_size` | `size` | `64M` | Maximum object size allowed to be stored in the writecache. | +| `workers_number` | `int` | `20` | Amount of background workers that move data from the writecache to the blobstor. | +| `max_batch_size` | `int` | `1000` | Maximum amount of small object `PUT` operations to perform in a single transaction. | +| `max_batch_delay` | `duration` | `10ms` | Maximum delay before a batch starts. | + + +# `node` section + +```yaml +node: + wallet: + path: /path/to/wallet.json + address: NcpJzXcSDrh5CCizf4K9Ro6w4t59J5LKzz + password: password + addresses: + - grpc://external.ip:8082 + attribute: + - "Price:11" + - "UN-LOCODE:RU MSK" + - "key:value" + relay: false + persistent_sessions: + path: /sessions + persistent_state: + path: /state + subnet: + exit_zero: false + entries: + - 123 + notification: + enabled: true + endpoint: tls://localhost:4222 + timeout: 6s + default_topic: topic + certificate: /path/to/cert.pem + key: /path/to/key.pem + ca: /path/to/ca.pem +``` + +| Parameter | Type | Default value | Description | +|-----------------------|---------------------------------------------------------------|---------------|-------------------------------------------------------------------------| +| `key` | `string` | | Path to the binary-encoded private key. | +| `wallet` | [Wallet config](#wallet-subsection) | | Wallet configuration. Has no effect if `key` is provided. | +| `addresses` | `[]string` | | Addresses advertised in the netmap. | +| `attribute` | `[]string` | | Node attributes as a list of key-value pairs in `:` format. | +| `relay` | `bool` | | Enable relay mode. | +| `persistent_sessions` | [Persistent sessions config](#persistent_sessions-subsection) | | Persistent session token store configuration. | +| `persistent_state` | [Persistent state config](#persistent_state-subsection) | | Persistent state configuration. | +| `subnet` | [Subnet config](#subnet-subsection) | | Subnet configuration. | +| `notification` | [Notification config](#notification-subsection) | | NATS configuration. | + + +## `wallet` subsection +N3 wallet configuration. + +| Parameter | Type | Default value | Description | +|------------|----------|---------------|------------------------------| +| `path` | `string` | | Path to the wallet file. | +| `address` | `string` | | Wallet address to use. | +| `password` | `string` | | Password to open the wallet. | + +## `persistent_sessions` subsection + +Contains persistent session token store configuration. By default sessions do not persist between restarts. + +| Parameter | Type | Default value | Description | +|-----------|----------|---------------|-----------------------| +| `path` | `string` | | Path to the database. | + +## `persistent_state` subsection +Configures persistent storage for auxiliary information, such as last seen block height. +It is used to correctly handle node restarts or crashes. + +| Parameter | Type | Default value | Description | +|-----------|----------|------------------------|------------------------| +| `path` | `string` | `.neofs-storage-state` | Path to the database. | + +## `subnet` subsection +This is an advanced section, use with caution. + +| Parameter | Type | Default value | Description | +|-------------|------------|---------------|------------------------------------------------------| +| `exit_zero` | `bool` | `false` | Exit from the default subnet. | +| `entries` | `[]uint32` | | List of non-default subnet ID this node belongs to. | + +## `notification` subsection +This is an advanced section, use with caution. + +| Parameter | Type | Default value | Description | +|-----------------|------------|-------------------|-------------------------------------------------------------------| +| `enabled` | `bool` | `false` | Flag to enable the service. | +| `endpoint` | `string` | | NATS endpoint to connect to. | +| `timeout` | `duration` | `5s` | Timeout for the object notification operation. | +| `default_topic` | `string` | node's public key | Default topic to use if an object has no corresponding attribute. | +| `certificate` | `string` | | Path to the client certificate. | +| `key` | `string` | | Path to the client key. | +| `ca` | `string` | | Override root CA used to verify server certificates. | + +# `apiclient` section +Configuration for the NeoFS API client used for communication with other NeoFS nodes. + +```yaml +apiclient: + dial_timeout: 15s +``` +| Parameter | Type | Default value | Description | +|--------------|----------|---------------|-----------------------------------------------------------------------| +| dial_timeout | duration | `5s` | Timeout for dialing connections to other storage or inner ring nodes. | + +# `policer` section + +Configuration for the Policer service. It ensures that object is stored according to the intended policy. + +```yaml +policer: + head_timeout: 15s +``` + +| Parameter | Type | Default value | Description | +|----------------|------------|---------------|----------------------------------------------| +| `head_timeout` | `duration` | `5s` | Timeout for performing the `HEAD` operation. | + +# `replicator` section + +Configuration for the Replicator service. + +```yaml +replicator: + put_timeout: 15s +``` + +| Parameter | Type | Default value | Description | +|---------------|------------|---------------|---------------------------------------------| +| `put_timeout` | `duration` | `5s` | Timeout for performing the `PUT` operation. | + +# `object` section +Contains pool sizes for object operations with remote nodes. + +```yaml +object: + put: + pool_size_remote: 100 +``` + +| Parameter | Type | Default value | Description | +|------------------------|-------|---------------|------------------------------------------------------------------------------------------------| +| `put.pool_size_remote` | `int` | `10` | Max pool size for performing remote `PUT` operations. Used by Policer and Replicator services. | \ No newline at end of file