Write-cache is flushing only within timeout CLI period #477

New issue

Closed

opened 2023-06-28 07:52:57 +00:00 by anikeev-yadro · 2 comments

anikeev-yadro commented

2023-06-28 07:52:57 +00:00

Member

Expected Behavior

Write-cache should continue flushing after CLI stopped due to timeout exeeded.

Current Behavior

Write-cache is flushing only within timeout CLI period.
After timeout exeeded CLI returned error rpc error: context deadline exceeded and write-cache stopped flushing.

Steps to Reproduce (for bugs)

1.Check that WC contains data

service@aanikeev-node1:~$ sudo du  /srv/frostfs/meta*|grep write_cache[0-9]*$
2807436 /srv/frostfs/meta0/write_cache3
1361252 /srv/frostfs/meta0/write_cache1
24      /srv/frostfs/meta0/write_cache0
2957540 /srv/frostfs/meta0/write_cache2
1396296 /srv/frostfs/meta0/write_cache4

2.Try to fush it

service@aanikeev-node1:~$ sudo frostfs-cli --endpoint 127.0.0.1:8091 -w /etc/frostfs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all
Enter password >
rpc error: context deadline exceeded

3.See that messeges about flush process appears in the log only within the CLI timeout

service@aanikeev-node1:~$ journalctl -f -u frostfs-storage --grep=cache
-- Journal begins at Tue 2023-06-27 11:23:17 UTC. --
Jun 28 07:48:19 aanikeev-node1 frostfs-node[179047]: 2023-06-28T07:48:19.158Z        info        log/log.go:13        local object storage operation        {"shard_id": "3aqRYb7F2bpETb48PFUyPs", "address": "EpcTWNjqw8Ws53fjhYA6GQxsfkHPuRvVUi3aPmQ2JXAd/ADfo3TKKWYg2GDqbCK5t6trSsYeV7pAKMN599vBPK1BT", "type": "write-cache", "op": "fstree DELETE"}
Jun 28 07:48:19 aanikeev-node1 frostfs-node[179047]: 2023-06-28T07:48:19.212Z        info        log/log.go:13        local object storage operation        {"shard_id": "3aqRYb7F2bpETb48PFUyPs", "address": "EpcTWNjqw8Ws53fjhYA6GQxsfkHPuRvVUi3aPmQ2JXAd/AEudJt48zzZZwUa1wQk76XCg6cjX885Js1n1LdWFNb4Y", "type": "write-cache", "op": "fstree DELETE"}
.
.
.
Jun 28 07:48:34 aanikeev-node1 frostfs-node[179047]: 2023-06-28T07:48:34.070Z        info        log/log.go:13        local object storage operation        {"shard_id": "3aqRYb7F2bpETb48PFUyPs", "address": "EpcTWNjqw8Ws53fjhYA6GQxsfkHPuRvVUi3aPmQ2JXAd/zRxojQg8sxBj83VyoSC4CyYmqVu9w8y9TNVW4JZTxmf", "type": "write-cache", "op": "fstree DELETE"}
Jun 28 07:48:34 aanikeev-node1 frostfs-node[179047]: 2023-06-28T07:48:34.123Z        info        log/log.go:13        local object storage operation        {"shard_id": "4G8Kx9AbWHoBvg9WpXzQvR", "address": "EpcTWNjqw8Ws53fjhYA6GQxsfkHPuRvVUi3aPmQ2JXAd/14KwEVu5qWdm63k72adpY2PYqqdWMSJCfVK9DU5QaRCg", "type": "write-cache", "op": "fstree DELETE"}

4.Check that some data was flushed from cache:

root@aanikeev-node1:~# sudo du  /srv/frostfs/meta*|grep write_cache[0-9]*$
2807436 /srv/frostfs/meta0/write_cache3
164     /srv/frostfs/meta0/write_cache1
24      /srv/frostfs/meta0/write_cache0
2952536 /srv/frostfs/meta0/write_cache2
1396296 /srv/frostfs/meta0/write_cache4

Regression

Yes

Version

FrostFS Storage node
Version: v0.0.1-682-g43d263c3
GoVersion: go1.19.7

Your Environment

Virtual
4 nodes

## Expected Behavior Write-cache should continue flushing after CLI stopped due to timeout exeeded. ## Current Behavior Write-cache is flushing only within timeout CLI period. After timeout exeeded CLI returned error `rpc error: context deadline exceeded` and write-cache stopped flushing. ## Steps to Reproduce (for bugs) 1.Check that WC contains data ``` service@aanikeev-node1:~$ sudo du /srv/frostfs/meta*|grep write_cache[0-9]*$ 2807436 /srv/frostfs/meta0/write_cache3 1361252 /srv/frostfs/meta0/write_cache1 24 /srv/frostfs/meta0/write_cache0 2957540 /srv/frostfs/meta0/write_cache2 1396296 /srv/frostfs/meta0/write_cache4 ``` 2.Try to fush it ``` service@aanikeev-node1:~$ sudo frostfs-cli --endpoint 127.0.0.1:8091 -w /etc/frostfs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all Enter password > rpc error: context deadline exceeded ``` 3.See that messeges about flush process appears in the log only within the CLI timeout ``` service@aanikeev-node1:~$ journalctl -f -u frostfs-storage --grep=cache -- Journal begins at Tue 2023-06-27 11:23:17 UTC. -- Jun 28 07:48:19 aanikeev-node1 frostfs-node[179047]: 2023-06-28T07:48:19.158Z info log/log.go:13 local object storage operation {"shard_id": "3aqRYb7F2bpETb48PFUyPs", "address": "EpcTWNjqw8Ws53fjhYA6GQxsfkHPuRvVUi3aPmQ2JXAd/ADfo3TKKWYg2GDqbCK5t6trSsYeV7pAKMN599vBPK1BT", "type": "write-cache", "op": "fstree DELETE"} Jun 28 07:48:19 aanikeev-node1 frostfs-node[179047]: 2023-06-28T07:48:19.212Z info log/log.go:13 local object storage operation {"shard_id": "3aqRYb7F2bpETb48PFUyPs", "address": "EpcTWNjqw8Ws53fjhYA6GQxsfkHPuRvVUi3aPmQ2JXAd/AEudJt48zzZZwUa1wQk76XCg6cjX885Js1n1LdWFNb4Y", "type": "write-cache", "op": "fstree DELETE"} . . . Jun 28 07:48:34 aanikeev-node1 frostfs-node[179047]: 2023-06-28T07:48:34.070Z info log/log.go:13 local object storage operation {"shard_id": "3aqRYb7F2bpETb48PFUyPs", "address": "EpcTWNjqw8Ws53fjhYA6GQxsfkHPuRvVUi3aPmQ2JXAd/zRxojQg8sxBj83VyoSC4CyYmqVu9w8y9TNVW4JZTxmf", "type": "write-cache", "op": "fstree DELETE"} Jun 28 07:48:34 aanikeev-node1 frostfs-node[179047]: 2023-06-28T07:48:34.123Z info log/log.go:13 local object storage operation {"shard_id": "4G8Kx9AbWHoBvg9WpXzQvR", "address": "EpcTWNjqw8Ws53fjhYA6GQxsfkHPuRvVUi3aPmQ2JXAd/14KwEVu5qWdm63k72adpY2PYqqdWMSJCfVK9DU5QaRCg", "type": "write-cache", "op": "fstree DELETE"} ``` 4.Check that some data was flushed from cache: ``` root@aanikeev-node1:~# sudo du /srv/frostfs/meta*|grep write_cache[0-9]*$ 2807436 /srv/frostfs/meta0/write_cache3 164 /srv/frostfs/meta0/write_cache1 24 /srv/frostfs/meta0/write_cache0 2952536 /srv/frostfs/meta0/write_cache2 1396296 /srv/frostfs/meta0/write_cache4 ``` ## Regression Yes ## Version ``` FrostFS Storage node Version: v0.0.1-682-g43d263c3 GoVersion: go1.19.7 ``` ## Your Environment Virtual 4 nodes