frostfs-node/docs/shard-modes.md

Shard modes description

List of modes

Each mode is characterized by two important properties:

1. Whether modifying operations are allowed.
2. Whether metabase and write-cache is available. The expected deployment scenario is to place both metabase and write-cache on an SSD drive thus these modes can be approximately described as no-SSD modes.

Mode	Description
read-write	Default mode, all operations are allowed.
read-only	Read-only mode, only read operations are allowed, metabase is available.
degraded	Degraded mode in which metabase and write-cache is disabled. It shouldn't be used at all, because metabase can contain important indices, such as LOCK objects info and modifying operation in this mode can lead to unexpected behaviour. The purpose of this mode is to allow PUT/DELETE operations without the metabase if really necessary.
degraded-read-only	Same as degraded, but with only read operations allowed. This mode is used during SSD replacement and/or when the metabase error counter exceeds threshold.
disabled	Currently used only in config file to temporarily disable a shard.

Shard and Component Status

Shard Mode	Metabase Mode	Blobstore Mode	Writecache Mode	Pilorama Mode	Blobovnicza Tree Mode	FSTree Mode
Read-Write	READ_WRITE	READ_WRITE	READ_WRITE	READ_WRITE	READ_WRITE	READ_WRITE
Read-Only	READ_ONLY	READ_ONLY	READ_ONLY	READ_ONLY	READ_ONLY	READ_ONLY
Degraded-Read-Write	CLOSED	READ_WRITE	CLOSED	CLOSED	READ_WRITE	READ_WRITE
Degraded-Read-Only	CLOSED	READ_ONLY	CLOSED	CLOSED	READ_ONLY	READ_ONLY

Transition order

Because each shard consists of multiple components changing its mode is not an atomic operation. Instead, each component changes its mode independently.

For transitions to read-write mode the order is:

1. metabase
2. blobstor
3. writecache
4. pilorama

For transitions to all other modes the order is:

1. writecache
2. blobstor
3. metabase
4. pilorama

The motivation is to avoid transient errors because write-cache can write to both blobstor and metabase. Thus, when we want to stop write operations, write-cache needs to change mode before them. On the other side, when we want to allow them, blobstor and metabase should be writable before write-cache is.

If anything goes wrong in the middle, the mode of some components can be different from the actual mode of a shard. However, all mode changing operations are idempotent.

Automatic mode changes

Shard can automatically switch to a read-only mode in 3 cases:

1. If the metabase was not available or couldn't be opened/initialized during shard startup.
2. If shard error counter exceeds threshold.
3. If the metabase couldn't be reopened during SIGHUP handling.

If switching to read-only is impossible, the shard will switch to degraded-read-only.

Detach shard

To detach a shard use frostfs-cli control shards detach command. This command removes the shards from the storage engine and closes all resources associated with the shards. Limitation: SIGHUP or storage node restart lead to detached shard will be again online.