Write-cache is not automatically flushing

anikeev-yadro commented

2023-06-28 08:56:01 +00:00

Expected Behavior

According new WC model it should be flush objects after small time after PUT to WC.

Current Behavior

Write-cache not automatically flushing (within load, after load stopped, within start/stop node).

Steps to Reproduce (for bugs)

1.Check size of WC

Jun 28 08:38:22 sudo du  /srv/frostfs/meta*|grep write_cache[0-9]*$
2807436 /srv/frostfs/meta0/write_cache3
164     /srv/frostfs/meta0/write_cache1
24      /srv/frostfs/meta0/write_cache0
2952536 /srv/frostfs/meta0/write_cache2
1396296 /srv/frostfs/meta0/write_cache4

2.Run k6 load

service@loader02:/etc/k6$ ./k6 run -e DURATION=1800 -e WRITERS=2 -e READERS=0 -e DELETERS=0 -e DELETE_AGE=10 -e REGISTRY_FILE=/var/tmp/grpc_reg_2806-1.json -e WRITE_OBJ_SIZE=5000 -e GRPC_ENDPOINTS=172.26.161.76:8080,172.26.161.64:8080,172.26.161.246:8080,172.26.161.224:8080 -e PREGEN_JSON=/var/tmp/grpc_2706-1.json -e STREAM_TIMEOUT=30 -e SLEEP_READ=0.2 -e SLEEP_WRITE=0.2 scenarios/grpc.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: scenarios/grpc.js
     output: -

  scenarios: (100.00%) 1 scenario, 2 max VUs, 30m5s max duration (incl. graceful stop):
           * write: 2 looping VUs for 30m0s (exec: obj_write, gracefulStop: 5s)

INFO[08:39:52] Pregenerated containers:       1              source=console
INFO[08:39:52] Pregenerated read object size: 20000 Kb       source=console
INFO[08:39:52] Pregenerated total objects:    500            source=console
INFO[08:39:52] Reading VUs:                   0              source=console
INFO[08:39:52] Writing VUs:                   2              source=console
INFO[08:39:52] Deleting VUs:                  0              source=console
INFO[08:39:52] Total VUs:                     2              source=console

running (00m13.1s), 0/2 VUs, 62 complete and 2 interrupted iterations
write ✗ [--------------------------------------] 2 VUs  00m13.1s/30m0s
     █ setup

     █ teardown

     data_received..............: 0 B    0 B/s
     data_sent..................: 317 MB 24 MB/s
     frostfs_obj_put_duration...: avg=188.02ms min=145.96ms med=182.8ms  max=255.85ms p(90)=223.5ms  p(95)=232.33ms
     frostfs_obj_put_total......: 62     4.731786/s
     iteration_duration.........: avg=401.01ms min=133.82µs med=400.16ms max=770.71ms p(90)=446.72ms p(95)=466.84ms
     iterations.................: 62     4.731786/s
     vus........................: 2      min=2      max=2
     vus_max....................: 2      min=2      max=2

3.Check that size of WC was grew

Jun 28 08:40:34 sudo du  /srv/frostfs/meta*|grep write_cache[0-9]*$
2827452 /srv/frostfs/meta0/write_cache3
5168    /srv/frostfs/meta0/write_cache1
60112   /srv/frostfs/meta0/write_cache0
2987568 /srv/frostfs/meta0/write_cache2
1411308 /srv/frostfs/meta0/write_cache4

4.Wait some time and check that size of WC did not decrease

root@aanikeev-node1:~# date; du  /srv/frostfs/meta*|grep write_cache[0-9]*$
Wed Jun 28 08:52:40 UTC 2023
2827452 /srv/frostfs/meta0/write_cache3
5168    /srv/frostfs/meta0/write_cache1
60112   /srv/frostfs/meta0/write_cache0
2987568 /srv/frostfs/meta0/write_cache2
1411308 /srv/frostfs/meta0/write_cache4

5.During #1-#4 I didn't see any messges in logs about WC flush

service@aanikeev-node1:~$ journalctl -f -u frostfs-storage --grep=cache
-- Journal begins at Tue 2023-06-27 11:23:17 UTC. --

Regression

yes

Version

FrostFS Storage node
Version: v0.0.1-682-g43d263c3
GoVersion: go1.19.7

Your Environment

Virtual
4 nodes

## Expected Behavior According new WC model it should be flush objects after small time after PUT to WC. ## Current Behavior Write-cache not automatically flushing (within load, after load stopped, within start/stop node). ## Steps to Reproduce (for bugs) 1.Check size of WC ``` Jun 28 08:38:22 sudo du /srv/frostfs/meta*|grep write_cache[0-9]*$ 2807436 /srv/frostfs/meta0/write_cache3 164 /srv/frostfs/meta0/write_cache1 24 /srv/frostfs/meta0/write_cache0 2952536 /srv/frostfs/meta0/write_cache2 1396296 /srv/frostfs/meta0/write_cache4 ``` 2.Run k6 load ``` service@loader02:/etc/k6$ ./k6 run -e DURATION=1800 -e WRITERS=2 -e READERS=0 -e DELETERS=0 -e DELETE_AGE=10 -e REGISTRY_FILE=/var/tmp/grpc_reg_2806-1.json -e WRITE_OBJ_SIZE=5000 -e GRPC_ENDPOINTS=172.26.161.76:8080,172.26.161.64:8080,172.26.161.246:8080,172.26.161.224:8080 -e PREGEN_JSON=/var/tmp/grpc_2706-1.json -e STREAM_TIMEOUT=30 -e SLEEP_READ=0.2 -e SLEEP_WRITE=0.2 scenarios/grpc.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io execution: local script: scenarios/grpc.js output: - scenarios: (100.00%) 1 scenario, 2 max VUs, 30m5s max duration (incl. graceful stop): * write: 2 looping VUs for 30m0s (exec: obj_write, gracefulStop: 5s) INFO[08:39:52] Pregenerated containers: 1 source=console INFO[08:39:52] Pregenerated read object size: 20000 Kb source=console INFO[08:39:52] Pregenerated total objects: 500 source=console INFO[08:39:52] Reading VUs: 0 source=console INFO[08:39:52] Writing VUs: 2 source=console INFO[08:39:52] Deleting VUs: 0 source=console INFO[08:39:52] Total VUs: 2 source=console running (00m13.1s), 0/2 VUs, 62 complete and 2 interrupted iterations write ✗ [--------------------------------------] 2 VUs 00m13.1s/30m0s █ setup █ teardown data_received..............: 0 B 0 B/s data_sent..................: 317 MB 24 MB/s frostfs_obj_put_duration...: avg=188.02ms min=145.96ms med=182.8ms max=255.85ms p(90)=223.5ms p(95)=232.33ms frostfs_obj_put_total......: 62 4.731786/s iteration_duration.........: avg=401.01ms min=133.82µs med=400.16ms max=770.71ms p(90)=446.72ms p(95)=466.84ms iterations.................: 62 4.731786/s vus........................: 2 min=2 max=2 vus_max....................: 2 min=2 max=2 ``` 3.Check that size of WC was grew ``` Jun 28 08:40:34 sudo du /srv/frostfs/meta*|grep write_cache[0-9]*$ 2827452 /srv/frostfs/meta0/write_cache3 5168 /srv/frostfs/meta0/write_cache1 60112 /srv/frostfs/meta0/write_cache0 2987568 /srv/frostfs/meta0/write_cache2 1411308 /srv/frostfs/meta0/write_cache4 ``` 4.Wait some time and check that size of WC did not decrease ``` root@aanikeev-node1:~# date; du /srv/frostfs/meta*|grep write_cache[0-9]*$ Wed Jun 28 08:52:40 UTC 2023 2827452 /srv/frostfs/meta0/write_cache3 5168 /srv/frostfs/meta0/write_cache1 60112 /srv/frostfs/meta0/write_cache0 2987568 /srv/frostfs/meta0/write_cache2 1411308 /srv/frostfs/meta0/write_cache4 ``` 5.During #1-#4 I didn't see any messges in logs about WC flush ``` service@aanikeev-node1:~$ journalctl -f -u frostfs-storage --grep=cache -- Journal begins at Tue 2023-06-27 11:23:17 UTC. -- ``` ## Regression yes ## Version ``` FrostFS Storage node Version: v0.0.1-682-g43d263c3 GoVersion: go1.19.7 ``` ## Your Environment Virtual 4 nodes

anikeev-yadro added the

bug

triage

labels 2023-06-28 08:56:01 +00:00

anikeev-yadro changed title from ~~Write-cache not automatically flushing~~ to Write-cache is not automatically flushing

2023-06-28 08:59:19 +00:00

ale64bit was assigned by fyrchik

2023-06-29 10:53:58 +00:00

ale64bit commented

2023-07-03 07:36:43 +00:00

Collaborator

@anikeev-yadro can you also post the config section relevant to the writecache?

anikeev-yadro commented

2023-07-03 09:07:20 +00:00

Poster

https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/config/example/node.yaml#L160

ale64bit commented

2023-07-03 09:46:43 +00:00

Collaborator

https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/config/example/node.yaml#L160

I meant the concrete one used for the test, not the example one.

> https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/config/example/node.yaml#L160 I meant the concrete one used for the test, not the example one.

anikeev-yadro commented

2023-07-03 10:52:01 +00:00

Poster

      writecache:
        capacity: 256gb
        enabled: true
        max_object_size: 8mb
        small_object_size: 128kb
        workers_number: 150

``` writecache: capacity: 256gb enabled: true max_object_size: 8mb small_object_size: 128kb workers_number: 150 ```

👍 1

ale64bit referenced a pull request that will close this issue

2023-07-13 09:15:52 +00:00

[#479] Fix writecache fstree flush premature ctx cancel #518