Write-cache didn't free up space after ran command `control shards flush-cache` and after cache was full filled #276

New issue

Closed

opened 2023-04-21 12:40:21 +00:00 by anikeev-yadro · 1 comment

anikeev-yadro commented

2023-04-21 12:40:21 +00:00

Member

Expected Behavior

Write-cache space should be free up if we use the following command control shards flush-cache.

Current Behavior

Write-cache space didn't free up after we used the following command control shards flush-cache.

Steps to Reproduce (for bugs)

1.Fill cluster while write-cache will be not full.

root@az:~date; du -h /srv/neofs/meta*|grep write_cache[0-9]*$*$
Thu Apr 20 15:09:52 UTC 2023
42G     /srv/neofs/meta0/write_cache0
42G     /srv/neofs/meta0/write_cache8
42G     /srv/neofs/meta0/write_cache6
42G     /srv/neofs/meta0/write_cache10
42G     /srv/neofs/meta0/write_cache2
42G     /srv/neofs/meta0/write_cache4
42G     /srv/neofs/meta1/write_cache7
42G     /srv/neofs/meta1/write_cache1
42G     /srv/neofs/meta1/write_cache9
42G     /srv/neofs/meta1/write_cache11
42G     /srv/neofs/meta1/write_cache3
42G     /srv/neofs/meta1/write_cache5

This is the all 256Gb according to config:

            writecache:
                capacity: 256gb

2.Flush cache manually using the following command

root@az:/home/service/frostfs-xk6# sudo neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all
Enter password >
Write-cache has been flushed.

Apr 21 11:42:53 az sudo[2461693]:     root : TTY=pts/2 ; PWD=/home/service/frostfs-xk6 ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all

3.Start load and see errors in log

Apr 21 12:28:22 az neofs-node[2415787]: 2023-04-21T12:28:22.487Z        debug        shard/put.go:61        can't put object to the write-cache, trying blobstor        {"shard_id": "LbU6FDosEmsn1MAhKkGRHb", "err": "no space left in the write cache"}

Version

NeoFS Storage node
Version: v0.35.0-64-g83d2e217-dirty
GoVersion: go1.18.4

Your Environment

## Expected Behavior Write-cache space should be free up if we use the following command `control shards flush-cache`. ## Current Behavior Write-cache space didn't free up after we used the following command `control shards flush-cache`. ## Steps to Reproduce (for bugs) 1.Fill cluster while write-cache will be not full. ``` root@az:~date; du -h /srv/neofs/meta*|grep write_cache[0-9]*$*$ Thu Apr 20 15:09:52 UTC 2023 42G /srv/neofs/meta0/write_cache0 42G /srv/neofs/meta0/write_cache8 42G /srv/neofs/meta0/write_cache6 42G /srv/neofs/meta0/write_cache10 42G /srv/neofs/meta0/write_cache2 42G /srv/neofs/meta0/write_cache4 42G /srv/neofs/meta1/write_cache7 42G /srv/neofs/meta1/write_cache1 42G /srv/neofs/meta1/write_cache9 42G /srv/neofs/meta1/write_cache11 42G /srv/neofs/meta1/write_cache3 42G /srv/neofs/meta1/write_cache5 ``` This is the all 256Gb according to config: ``` writecache: capacity: 256gb ``` 2.Flush cache manually using the following command ``` root@az:/home/service/frostfs-xk6# sudo neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all Enter password > Write-cache has been flushed. ``` ``` Apr 21 11:42:53 az sudo[2461693]: root : TTY=pts/2 ; PWD=/home/service/frostfs-xk6 ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all ``` 3.Start load and see errors in log ``` Apr 21 12:28:22 az neofs-node[2415787]: 2023-04-21T12:28:22.487Z debug shard/put.go:61 can't put object to the write-cache, trying blobstor {"shard_id": "LbU6FDosEmsn1MAhKkGRHb", "err": "no space left in the write cache"} ``` ## Version ``` NeoFS Storage node Version: v0.35.0-64-g83d2e217-dirty GoVersion: go1.18.4 ``` ## Your Environment HW

anikeev-yadro added the

triage

label 2023-04-21 12:40:21 +00:00

anikeev-yadro changed title from ~~Write-cache didn't free up space after command control shards flush-cache~~ to Write-cache didn't free up space after command control shards flush-cache and after cache was full filled

2023-04-21 12:40:52 +00:00

anikeev-yadro changed title from ~~Write-cache didn't free up space after command control shards flush-cache and after cache was full filled~~ to Write-cache didn't free up space after ran command control shards flush-cache and after cache was full filled

2023-04-21 12:41:48 +00:00