Write-cache not automatically flushing after full filled. #275

New Issue

Closed

opened 2023-04-21 06:47:14 +00:00 by anikeev-yadro · 1 comment

anikeev-yadro commented

2023-04-21 06:47:14 +00:00

Expected Behavior

Write-cache should flush old data for get new data.

Current Behavior

Write-cache not flushing after full filled.

Steps to Reproduce (for bugs)

1.Fill cluster while write-cache will be not full.

root@az:~date; du -h /srv/neofs/meta*|grep write_cache[0-9]*$*$
Thu Apr 20 15:09:52 UTC 2023
42G     /srv/neofs/meta0/write_cache0
42G     /srv/neofs/meta0/write_cache8
42G     /srv/neofs/meta0/write_cache6
42G     /srv/neofs/meta0/write_cache10
42G     /srv/neofs/meta0/write_cache2
42G     /srv/neofs/meta0/write_cache4
42G     /srv/neofs/meta1/write_cache7
42G     /srv/neofs/meta1/write_cache1
42G     /srv/neofs/meta1/write_cache9
42G     /srv/neofs/meta1/write_cache11
42G     /srv/neofs/meta1/write_cache3
42G     /srv/neofs/meta1/write_cache5

This is the all 256Gb according to config:

            writecache:
                capacity: 256gb

2.Started load on the node AZ:

/home/service/xk6-frostfs-alex/k6 run -e DURATION=600 -e WRITERS=50 -e READERS=50 -e DELETERS=0 -e DELETE_AGE=10 -e REGISTRY_FILE=/var/data/tmp/s3_reg_200423-1000-2.json -e WRITE_OBJ_SIZE=1000 -e S3_ENDPOINTS=10.78.70.118:8084 -e PREGEN_JSON=/var/data/tmp/s3_200423-1000.json -e STREAM_TIMEOUT=60 -e SLEEP_READ=0.2 -e SLEEP_WRITE=0.2 scenarios/s3.js

3.Saw errors in log:

Apr 20 15:08:52 az neofs-node[8517]: 2023-04-20T15:08:52.525Z        debug        shard/put.go:61        can't put object to the write-cache, trying blobstor        {"shard_id": "54Lrxipanm3rZububmCcMX", "err": "no space left in the write cache"}

4.But in the same time I didn't saw any messages about PUT object to the write-cache. All objects was put in the fstree.
5.Tried to flush-cache manually

root@az:~# sudo neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json --config /tmp/storage.yaml control shards                                                                                 flush-cache --all
Enter password >
rpc error: context deadline exceeded

Apr 20 15:21:18 az sudo[1924404]:     root : TTY=pts/2 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all

6.I have started the same load 15 hours later and saw the same errors in log. And I didn't see any messages about PUT object to the write-cache.

Apr 21 06:20:45 az neofs-node[8517]: 2023-04-21T06:20:45.515Z        debug        shard/put.go:61        can't put object to the write-cache, trying blobstor        {"shard_id": "7fjAWN4euDPZeySdZ1niV1", "err": "no space left in the write cache"}

Version

NeoFS Storage node
Version: v0.35.0-64-g83d2e217-dirty
GoVersion: go1.18.4

Your Environment

## Expected Behavior Write-cache should flush old data for get new data. ## Current Behavior Write-cache not flushing after full filled. ## Steps to Reproduce (for bugs) 1.Fill cluster while write-cache will be not full. ``` root@az:~date; du -h /srv/neofs/meta*|grep write_cache[0-9]*$*$ Thu Apr 20 15:09:52 UTC 2023 42G /srv/neofs/meta0/write_cache0 42G /srv/neofs/meta0/write_cache8 42G /srv/neofs/meta0/write_cache6 42G /srv/neofs/meta0/write_cache10 42G /srv/neofs/meta0/write_cache2 42G /srv/neofs/meta0/write_cache4 42G /srv/neofs/meta1/write_cache7 42G /srv/neofs/meta1/write_cache1 42G /srv/neofs/meta1/write_cache9 42G /srv/neofs/meta1/write_cache11 42G /srv/neofs/meta1/write_cache3 42G /srv/neofs/meta1/write_cache5 ``` This is the all 256Gb according to config: ``` writecache: capacity: 256gb ``` 2.Started load on the node AZ: ``` /home/service/xk6-frostfs-alex/k6 run -e DURATION=600 -e WRITERS=50 -e READERS=50 -e DELETERS=0 -e DELETE_AGE=10 -e REGISTRY_FILE=/var/data/tmp/s3_reg_200423-1000-2.json -e WRITE_OBJ_SIZE=1000 -e S3_ENDPOINTS=10.78.70.118:8084 -e PREGEN_JSON=/var/data/tmp/s3_200423-1000.json -e STREAM_TIMEOUT=60 -e SLEEP_READ=0.2 -e SLEEP_WRITE=0.2 scenarios/s3.js ``` 3.Saw errors in log: ``` Apr 20 15:08:52 az neofs-node[8517]: 2023-04-20T15:08:52.525Z debug shard/put.go:61 can't put object to the write-cache, trying blobstor {"shard_id": "54Lrxipanm3rZububmCcMX", "err": "no space left in the write cache"} ``` 4.But in the same time I didn't saw any messages about PUT object to the write-cache. All objects was put in the fstree. 5.Tried to flush-cache manually ``` root@az:~# sudo neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all Enter password > rpc error: context deadline exceeded ``` ``` Apr 20 15:21:18 az sudo[1924404]: root : TTY=pts/2 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all ``` 6.I have started the same load 15 hours later and saw the same errors in log. And I didn't see any messages about PUT object to the write-cache. ``` Apr 21 06:20:45 az neofs-node[8517]: 2023-04-21T06:20:45.515Z debug shard/put.go:61 can't put object to the write-cache, trying blobstor {"shard_id": "7fjAWN4euDPZeySdZ1niV1", "err": "no space left in the write cache"} ``` ## Version ``` NeoFS Storage node Version: v0.35.0-64-g83d2e217-dirty GoVersion: go1.18.4 ``` ## Your Environment HW