TrueCloudLab/frostfs-node
20
1
Fork 27
Code Issues 100 Pull requests 4 Projects Releases 1 Activity Actions

Try using badger for the write-cache #421

New issue
Closed
opened 2023-06-02 11:43:11 +00:00 by fyrchik · 9 comments
fyrchik commented 2023-06-02 11:43:11 +00:00
Owner
Copy link

Last time we touched it it was losing data on hard resets.
The new release can be better, though.

Last time we touched it it was losing data on hard resets. The new release can be better, though.
fyrchik added this to the vNext milestone 2023-06-02 11:43:11 +00:00
fyrchik added the
frostfs-node
triage
 labels 2023-06-02 11:43:11 +00:00
ale64bit commented 2023-06-05 10:58:14 +00:00
Member
Copy link

Last time we touched it it was losing data on hard resets.

Would be good to clarify how exactly it was losing data. I suppose you mean that when the process was stopped, some writes were lost because they were not flushed to disk yet? If that's the case, it's kinda expected. LevelDB has a sync mode for that purpose.

If not, was there an open issue about it where we can track whether they did something about it?

> Last time we touched it it was losing data on hard resets. Would be good to clarify how exactly it was losing data. I suppose you mean that when the process was stopped, some writes were lost because they were not flushed to disk yet? If that's the case, it's kinda expected. LevelDB has a [sync](https://github.com/google/leveldb/blob/main/doc/index.md#synchronous-writes) mode for that purpose. If not, was there an open issue about it where we can track whether they did something about it?
ale64bit self-assigned this 2023-06-14 07:08:14 +00:00
ale64bit commented 2023-06-30 12:34:07 +00:00
Member
Copy link

Here are the write load test results for bbolt-backed writecache (from master branch) and the badger-backed writecache (#462):

================================================================================
T=1min S=8KiB VUs=1

--------------------------------------------------------------------------------
master
     data_sent................: 15 MB 254 kB/s
     iteration_duration.......: avg=32.25ms min=4.02µs  med=32.44ms max=44.76ms p(90)=32.91ms p(95)=33.07ms
     iterations...............: 1857  30.947383/s
     local_obj_put_duration...: avg=31.88ms min=17.09ms med=32.08ms max=44.35ms p(90)=32.46ms p(95)=32.61ms
     local_obj_put_total......: 1857  30.947383/s

--------------------------------------------------------------------------------
badger
     data_sent................: 22 MB 369 kB/s
     iteration_duration.......: avg=22.15ms min=3.83µs med=21.69ms max=56.28ms p(90)=22.59ms p(95)=29ms   
     iterations...............: 2704  45.058019/s
     local_obj_put_duration...: avg=21.79ms min=2.85ms med=21.33ms max=55.83ms p(90)=22.18ms p(95)=28.56ms
     local_obj_put_total......: 2704  45.058019/s

================================================================================
T=1min S=8KiB VUs=10

--------------------------------------------------------------------------------
master
     data_sent................: 131 MB 2.2 MB/s
     iteration_duration.......: avg=37.26ms min=4.35µs  med=36.19ms max=82.7ms  p(90)=38.55ms p(95)=42.56ms
     iterations...............: 16090  268.130094/s
     local_obj_put_duration...: avg=36.86ms min=22.27ms med=35.82ms max=82.37ms p(90)=38.05ms p(95)=41.86ms
     local_obj_put_total......: 16090  268.130094/s

--------------------------------------------------------------------------------
badger
     data_sent................: 181 MB 3.0 MB/s
     iteration_duration.......: avg=26.94ms min=3.22µs med=24.97ms max=106.61ms p(90)=39.37ms p(95)=43.46ms
     iterations...............: 22256  370.896659/s
     local_obj_put_duration...: avg=26.58ms min=2.76ms med=24.64ms max=106.2ms  p(90)=38.97ms p(95)=43.05ms
     local_obj_put_total......: 22256  370.896659/s

================================================================================
T=1min S=8KiB VUs=100

--------------------------------------------------------------------------------
master
     data_sent................: 625 MB 10 MB/s
     iteration_duration.......: avg=78.42ms min=3.46µs  med=72.39ms max=201.41ms p(90)=107.4ms  p(95)=117.26ms
     iterations...............: 76500  1274.475346/s
     local_obj_put_duration...: avg=77.95ms min=33.71ms med=71.97ms max=201.08ms p(90)=106.89ms p(95)=116.76ms
     local_obj_put_total......: 76500  1274.475346/s

--------------------------------------------------------------------------------
badger
     data_sent................: 747 MB 12 MB/s
     iteration_duration.......: avg=65.55ms min=1.82µs med=59.54ms max=401.81ms p(90)=80.99ms p(95)=93.62ms
     iterations...............: 91542  1524.991206/s
     local_obj_put_duration...: avg=65.19ms min=2.93ms med=59.21ms max=401.52ms p(90)=80.6ms  p(95)=93.26ms
     local_obj_put_total......: 91542  1524.991206/s

================================================================================
T=1min S=1KiB VUs=1

--------------------------------------------------------------------------------
master
     data_sent................: 1.9 MB 32 kB/s
     iteration_duration.......: avg=32.19ms min=2.71µs  med=32.13ms max=41.99ms p(90)=32.81ms p(95)=32.94ms
     iterations...............: 1861   31.005749/s
     local_obj_put_duration...: avg=31.85ms min=20.35ms med=31.74ms max=41.74ms p(90)=32.4ms  p(95)=32.51ms
     local_obj_put_total......: 1861   31.005749/s

--------------------------------------------------------------------------------
badger
     data_sent................: 2.8 MB 47 kB/s
     iteration_duration.......: avg=21.85ms min=4.56µs med=21.64ms max=44.01ms p(90)=22.46ms p(95)=24.44ms
     iterations...............: 2742   45.682231/s
     local_obj_put_duration...: avg=21.51ms min=3.1ms  med=21.3ms  max=43.69ms p(90)=22.05ms p(95)=23.96ms
     local_obj_put_total......: 2742   45.682231/s

================================================================================
T=1min S=1KiB VUs=10

--------------------------------------------------------------------------------
master
     data_sent................: 17 MB 284 kB/s
     iteration_duration.......: avg=35.71ms min=3.06µs  med=35.29ms max=68.26ms p(90)=36.97ms p(95)=38.58ms
     iterations...............: 16790 279.785838/s
     local_obj_put_duration...: avg=35.36ms min=15.78ms med=34.95ms max=67.96ms p(90)=36.57ms p(95)=38.05ms
     local_obj_put_total......: 16790 279.785838/s

--------------------------------------------------------------------------------
badger
     data_sent................: 24 MB 405 kB/s
     iteration_duration.......: avg=25.1ms  min=3.65µs med=24.86ms max=93.24ms p(90)=30.58ms p(95)=34.53ms
     iterations...............: 23894 398.064865/s
     local_obj_put_duration...: avg=24.79ms min=2.84ms med=24.57ms max=92.96ms p(90)=30.27ms p(95)=34.21ms
     local_obj_put_total......: 23894 398.064865/s

================================================================================
T=1min S=1KiB VUs=100

--------------------------------------------------------------------------------
master
     data_sent................: 92 MB 1.5 MB/s
     iteration_duration.......: avg=66.42ms min=4.5µs   med=64.68ms max=115.54ms p(90)=79.26ms p(95)=86.23ms
     iterations...............: 90379 1504.619563/s
     local_obj_put_duration...: avg=65.91ms min=21.68ms med=64.24ms max=114.61ms p(90)=78.62ms p(95)=85.66ms
     local_obj_put_total......: 90379 1504.619563/s

--------------------------------------------------------------------------------
badger
     data_sent................: 118 MB 2.0 MB/s
     iteration_duration.......: avg=52.07ms min=3.34µs med=51.82ms max=221.47ms p(90)=61.5ms  p(95)=65.97ms
     iterations...............: 115247 1919.433472/s
     local_obj_put_duration...: avg=51.76ms min=3.93ms med=51.52ms max=221.22ms p(90)=61.17ms p(95)=65.64ms
     local_obj_put_total......: 115247 1919.433472/s

================================================================================
T=10min S=64KiB VUs=1

--------------------------------------------------------------------------------
master
     data_sent................: 1.2 GB 1.9 MB/s
     iteration_duration.......: avg=33.63ms min=3.89µs  med=33.36ms max=62.02ms p(90)=34.19ms p(95)=34.57ms
     iterations...............: 17825  29.706705/s
     local_obj_put_duration...: avg=32.96ms min=13.16ms med=32.66ms max=61.35ms p(90)=33.48ms p(95)=33.79ms
     local_obj_put_total......: 17825  29.706705/s

--------------------------------------------------------------------------------
badger
     data_sent................: 944 MB 1.6 MB/s
     iteration_duration.......: avg=41.64ms min=3.76µs med=22.49ms max=584.13ms p(90)=112.86ms p(95)=130.26ms
     iterations...............: 14402  23.996169/s
     local_obj_put_duration...: avg=40.96ms min=1.53ms med=21.88ms max=582.77ms p(90)=112.03ms p(95)=129.42ms
     local_obj_put_total......: 14402  23.996169/s

================================================================================
T=10min S=4KiB VUs=10

--------------------------------------------------------------------------------
master
     data_sent................: 649 MB 1.1 MB/s
     iteration_duration.......: avg=37.5ms  min=3.7µs   med=36.46ms max=87.95ms p(90)=38.65ms p(95)=42.31ms
     iterations...............: 159890 266.479191/s
     local_obj_put_duration...: avg=37.14ms min=16.79ms med=36.12ms max=87.55ms p(90)=38.24ms p(95)=41.66ms
     local_obj_put_total......: 159890 266.479191/s

--------------------------------------------------------------------------------
badger
     data_sent................: 871 MB 1.5 MB/s
     iteration_duration.......: avg=27.97ms min=4.72µs med=28.76ms max=437.07ms p(90)=38.68ms p(95)=42.82ms
     iterations...............: 214360 357.263512/s
     local_obj_put_duration...: avg=27.66ms min=2.3ms  med=28.45ms max=436.48ms p(90)=38.37ms p(95)=42.5ms 
     local_obj_put_total......: 214360 357.263512/s

================================================================================
T=5min S=1KiB VUs=100

--------------------------------------------------------------------------------
master
     data_sent................: 416 MB 1.4 MB/s
     iteration_duration.......: avg=73.68ms min=3.35µs  med=70.25ms max=161.51ms p(90)=93.19ms p(95)=101.2ms 
     iterations...............: 407023 1356.656495/s
     local_obj_put_duration...: avg=73.25ms min=33.74ms med=69.87ms max=161.33ms p(90)=92.64ms p(95)=100.71ms
     local_obj_put_total......: 407023 1356.656495/s

--------------------------------------------------------------------------------
badger
     data_sent................: 511 MB 1.7 MB/s
     iteration_duration.......: avg=59.97ms min=2.62µs med=57.76ms max=478.98ms p(90)=68.86ms p(95)=74.31ms
     iterations...............: 500164 1666.901388/s
     local_obj_put_duration...: avg=59.65ms min=3.14ms med=57.47ms max=477.73ms p(90)=68.51ms p(95)=73.88ms
     local_obj_put_total......: 500164 1666.901388/s
Here are the write load test results for bbolt-backed writecache (from `master` branch) and the badger-backed writecache (#462): ``` ================================================================================ T=1min S=8KiB VUs=1 -------------------------------------------------------------------------------- master data_sent................: 15 MB 254 kB/s iteration_duration.......: avg=32.25ms min=4.02µs med=32.44ms max=44.76ms p(90)=32.91ms p(95)=33.07ms iterations...............: 1857 30.947383/s local_obj_put_duration...: avg=31.88ms min=17.09ms med=32.08ms max=44.35ms p(90)=32.46ms p(95)=32.61ms local_obj_put_total......: 1857 30.947383/s -------------------------------------------------------------------------------- badger data_sent................: 22 MB 369 kB/s iteration_duration.......: avg=22.15ms min=3.83µs med=21.69ms max=56.28ms p(90)=22.59ms p(95)=29ms iterations...............: 2704 45.058019/s local_obj_put_duration...: avg=21.79ms min=2.85ms med=21.33ms max=55.83ms p(90)=22.18ms p(95)=28.56ms local_obj_put_total......: 2704 45.058019/s ================================================================================ T=1min S=8KiB VUs=10 -------------------------------------------------------------------------------- master data_sent................: 131 MB 2.2 MB/s iteration_duration.......: avg=37.26ms min=4.35µs med=36.19ms max=82.7ms p(90)=38.55ms p(95)=42.56ms iterations...............: 16090 268.130094/s local_obj_put_duration...: avg=36.86ms min=22.27ms med=35.82ms max=82.37ms p(90)=38.05ms p(95)=41.86ms local_obj_put_total......: 16090 268.130094/s -------------------------------------------------------------------------------- badger data_sent................: 181 MB 3.0 MB/s iteration_duration.......: avg=26.94ms min=3.22µs med=24.97ms max=106.61ms p(90)=39.37ms p(95)=43.46ms iterations...............: 22256 370.896659/s local_obj_put_duration...: avg=26.58ms min=2.76ms med=24.64ms max=106.2ms p(90)=38.97ms p(95)=43.05ms local_obj_put_total......: 22256 370.896659/s ================================================================================ T=1min S=8KiB VUs=100 -------------------------------------------------------------------------------- master data_sent................: 625 MB 10 MB/s iteration_duration.......: avg=78.42ms min=3.46µs med=72.39ms max=201.41ms p(90)=107.4ms p(95)=117.26ms iterations...............: 76500 1274.475346/s local_obj_put_duration...: avg=77.95ms min=33.71ms med=71.97ms max=201.08ms p(90)=106.89ms p(95)=116.76ms local_obj_put_total......: 76500 1274.475346/s -------------------------------------------------------------------------------- badger data_sent................: 747 MB 12 MB/s iteration_duration.......: avg=65.55ms min=1.82µs med=59.54ms max=401.81ms p(90)=80.99ms p(95)=93.62ms iterations...............: 91542 1524.991206/s local_obj_put_duration...: avg=65.19ms min=2.93ms med=59.21ms max=401.52ms p(90)=80.6ms p(95)=93.26ms local_obj_put_total......: 91542 1524.991206/s ================================================================================ T=1min S=1KiB VUs=1 -------------------------------------------------------------------------------- master data_sent................: 1.9 MB 32 kB/s iteration_duration.......: avg=32.19ms min=2.71µs med=32.13ms max=41.99ms p(90)=32.81ms p(95)=32.94ms iterations...............: 1861 31.005749/s local_obj_put_duration...: avg=31.85ms min=20.35ms med=31.74ms max=41.74ms p(90)=32.4ms p(95)=32.51ms local_obj_put_total......: 1861 31.005749/s -------------------------------------------------------------------------------- badger data_sent................: 2.8 MB 47 kB/s iteration_duration.......: avg=21.85ms min=4.56µs med=21.64ms max=44.01ms p(90)=22.46ms p(95)=24.44ms iterations...............: 2742 45.682231/s local_obj_put_duration...: avg=21.51ms min=3.1ms med=21.3ms max=43.69ms p(90)=22.05ms p(95)=23.96ms local_obj_put_total......: 2742 45.682231/s ================================================================================ T=1min S=1KiB VUs=10 -------------------------------------------------------------------------------- master data_sent................: 17 MB 284 kB/s iteration_duration.......: avg=35.71ms min=3.06µs med=35.29ms max=68.26ms p(90)=36.97ms p(95)=38.58ms iterations...............: 16790 279.785838/s local_obj_put_duration...: avg=35.36ms min=15.78ms med=34.95ms max=67.96ms p(90)=36.57ms p(95)=38.05ms local_obj_put_total......: 16790 279.785838/s -------------------------------------------------------------------------------- badger data_sent................: 24 MB 405 kB/s iteration_duration.......: avg=25.1ms min=3.65µs med=24.86ms max=93.24ms p(90)=30.58ms p(95)=34.53ms iterations...............: 23894 398.064865/s local_obj_put_duration...: avg=24.79ms min=2.84ms med=24.57ms max=92.96ms p(90)=30.27ms p(95)=34.21ms local_obj_put_total......: 23894 398.064865/s ================================================================================ T=1min S=1KiB VUs=100 -------------------------------------------------------------------------------- master data_sent................: 92 MB 1.5 MB/s iteration_duration.......: avg=66.42ms min=4.5µs med=64.68ms max=115.54ms p(90)=79.26ms p(95)=86.23ms iterations...............: 90379 1504.619563/s local_obj_put_duration...: avg=65.91ms min=21.68ms med=64.24ms max=114.61ms p(90)=78.62ms p(95)=85.66ms local_obj_put_total......: 90379 1504.619563/s -------------------------------------------------------------------------------- badger data_sent................: 118 MB 2.0 MB/s iteration_duration.......: avg=52.07ms min=3.34µs med=51.82ms max=221.47ms p(90)=61.5ms p(95)=65.97ms iterations...............: 115247 1919.433472/s local_obj_put_duration...: avg=51.76ms min=3.93ms med=51.52ms max=221.22ms p(90)=61.17ms p(95)=65.64ms local_obj_put_total......: 115247 1919.433472/s ================================================================================ T=10min S=64KiB VUs=1 -------------------------------------------------------------------------------- master data_sent................: 1.2 GB 1.9 MB/s iteration_duration.......: avg=33.63ms min=3.89µs med=33.36ms max=62.02ms p(90)=34.19ms p(95)=34.57ms iterations...............: 17825 29.706705/s local_obj_put_duration...: avg=32.96ms min=13.16ms med=32.66ms max=61.35ms p(90)=33.48ms p(95)=33.79ms local_obj_put_total......: 17825 29.706705/s -------------------------------------------------------------------------------- badger data_sent................: 944 MB 1.6 MB/s iteration_duration.......: avg=41.64ms min=3.76µs med=22.49ms max=584.13ms p(90)=112.86ms p(95)=130.26ms iterations...............: 14402 23.996169/s local_obj_put_duration...: avg=40.96ms min=1.53ms med=21.88ms max=582.77ms p(90)=112.03ms p(95)=129.42ms local_obj_put_total......: 14402 23.996169/s ================================================================================ T=10min S=4KiB VUs=10 -------------------------------------------------------------------------------- master data_sent................: 649 MB 1.1 MB/s iteration_duration.......: avg=37.5ms min=3.7µs med=36.46ms max=87.95ms p(90)=38.65ms p(95)=42.31ms iterations...............: 159890 266.479191/s local_obj_put_duration...: avg=37.14ms min=16.79ms med=36.12ms max=87.55ms p(90)=38.24ms p(95)=41.66ms local_obj_put_total......: 159890 266.479191/s -------------------------------------------------------------------------------- badger data_sent................: 871 MB 1.5 MB/s iteration_duration.......: avg=27.97ms min=4.72µs med=28.76ms max=437.07ms p(90)=38.68ms p(95)=42.82ms iterations...............: 214360 357.263512/s local_obj_put_duration...: avg=27.66ms min=2.3ms med=28.45ms max=436.48ms p(90)=38.37ms p(95)=42.5ms local_obj_put_total......: 214360 357.263512/s ================================================================================ T=5min S=1KiB VUs=100 -------------------------------------------------------------------------------- master data_sent................: 416 MB 1.4 MB/s iteration_duration.......: avg=73.68ms min=3.35µs med=70.25ms max=161.51ms p(90)=93.19ms p(95)=101.2ms iterations...............: 407023 1356.656495/s local_obj_put_duration...: avg=73.25ms min=33.74ms med=69.87ms max=161.33ms p(90)=92.64ms p(95)=100.71ms local_obj_put_total......: 407023 1356.656495/s -------------------------------------------------------------------------------- badger data_sent................: 511 MB 1.7 MB/s iteration_duration.......: avg=59.97ms min=2.62µs med=57.76ms max=478.98ms p(90)=68.86ms p(95)=74.31ms iterations...............: 500164 1666.901388/s local_obj_put_duration...: avg=59.65ms min=3.14ms med=57.47ms max=477.73ms p(90)=68.51ms p(95)=73.88ms local_obj_put_total......: 500164 1666.901388/s ```
fyrchik commented 2023-06-30 12:39:43 +00:00
Author
Owner
Copy link

Badger seems to be worse for the 64KiB case.
Can you explain this?
The median is lower, but p(90) is significantly higher, I am wondering why: for 10-min test it seems significant.

Badger seems to be worse for the 64KiB case. Can you explain this? The median is lower, but p(90) is significantly higher, I am wondering why: for 10-min test it seems significant.
ale64bit commented 2023-06-30 13:11:08 +00:00
Member
Copy link

Badger seems to be worse for the 64KiB case.
Can you explain this?
The median is lower, but p(90) is significantly higher, I am wondering why: for 10-min test it seems significant.

I think it's not due to the object size specifically, but about VU=1. It basically causes it to write serially, which is not that fast by design (more info here).

But tests with more VUs run out of disk space in the test hardware.

> Badger seems to be worse for the 64KiB case. > Can you explain this? > The median is lower, but p(90) is significantly higher, I am wondering why: for 10-min test it seems significant. I think it's not due to the object size specifically, but about VU=1. It basically causes it to write serially, which is not that fast by design (more info [here](https://github.com/dgraph-io/badger/issues/869)). But tests with more VUs run out of disk space in the test hardware.
ale64bit commented 2023-07-03 06:34:36 +00:00
Member
Copy link

I reran the tests using the correct volumes:

================================================================================
T=1min S=8KiB VUs=1

--------------------------------------------------------------------------------
master
     data_sent................: 16 MB 258 kB/s
     iteration_duration.......: avg=31.7ms  min=3.86µs  med=32.25ms max=47.26ms p(90)=32.89ms p(95)=33.05ms
     iterations...............: 1890  31.490094/s
     local_obj_put_duration...: avg=31.33ms min=19.24ms med=31.84ms max=46.42ms p(90)=32.47ms p(95)=32.63ms
     local_obj_put_total......: 1890  31.490094/s

--------------------------------------------------------------------------------
badger
     data_sent................: 23 MB 374 kB/s
     iteration_duration.......: avg=21.85ms min=3.31µs  med=21.75ms max=43.12ms p(90)=22.46ms p(95)=22.57ms
     iterations...............: 2741  45.677653/s
     local_obj_put_duration...: avg=21.49ms min=17.19ms med=21.39ms max=42.82ms p(90)=22.04ms p(95)=22.16ms
     local_obj_put_total......: 2741  45.677653/s

================================================================================
T=1min S=8KiB VUs=10

--------------------------------------------------------------------------------
master
     data_sent................: 138 MB 2.3 MB/s
     iteration_duration.......: avg=35.4ms  min=4.46µs  med=37.16ms max=52.91ms p(90)=38.81ms p(95)=39.6ms 
     iterations...............: 16940  282.264017/s
     local_obj_put_duration...: avg=35.04ms min=16.32ms med=36.81ms max=52.64ms p(90)=38.47ms p(95)=39.23ms
     local_obj_put_total......: 16940  282.264017/s

--------------------------------------------------------------------------------
badger
     data_sent................: 193 MB 3.2 MB/s
     iteration_duration.......: avg=25.29ms min=3.49µs  med=25.18ms max=113.74ms p(90)=26.61ms p(95)=27.26ms
     iterations...............: 23705  395.047146/s
     local_obj_put_duration...: avg=24.95ms min=15.85ms med=24.85ms max=113.38ms p(90)=26.28ms p(95)=26.91ms
     local_obj_put_total......: 23705  395.047146/s

================================================================================
T=1min S=8KiB VUs=100

--------------------------------------------------------------------------------
master
     data_sent................: 611 MB 10 MB/s
     iteration_duration.......: avg=80.23ms min=3.85µs med=81.47ms max=121.99ms p(90)=92.87ms p(95)=97ms   
     iterations...............: 74809  1245.80559/s
     local_obj_put_duration...: avg=79.76ms min=32.5ms med=81.02ms max=121.59ms p(90)=92.45ms p(95)=96.57ms
     local_obj_put_total......: 74809  1245.80559/s

--------------------------------------------------------------------------------
badger
     data_sent................: 873 MB 15 MB/s
     iteration_duration.......: avg=56.17ms min=2.25µs med=54.23ms max=302.9ms  p(90)=64.14ms p(95)=74.92ms
     iterations...............: 106811 1779.295693/s
     local_obj_put_duration...: avg=55.81ms min=2.07ms med=53.87ms max=302.51ms p(90)=63.77ms p(95)=74.52ms
     local_obj_put_total......: 106811 1779.295693/s

================================================================================
T=5min S=1KiB VUs=1

--------------------------------------------------------------------------------
master
     data_sent................: 9.7 MB 32 kB/s
     iteration_duration.......: avg=31.53ms min=4.4µs   med=32.16ms max=44.83ms p(90)=32.92ms p(95)=33.08ms
     iterations...............: 9504   31.679499/s
     local_obj_put_duration...: avg=31.17ms min=15.69ms med=31.78ms max=43.66ms p(90)=32.54ms p(95)=32.7ms 
     local_obj_put_total......: 9504   31.679499/s

--------------------------------------------------------------------------------
badger
     data_sent................: 14 MB 47 kB/s
     iteration_duration.......: avg=21.96ms min=2.59µs  med=21.93ms max=32.09ms p(90)=22.51ms p(95)=22.64ms
     iterations...............: 13646 45.485952/s
     local_obj_put_duration...: avg=21.62ms min=15.29ms med=21.6ms  max=31.51ms p(90)=22.15ms p(95)=22.27ms
     local_obj_put_total......: 13646 45.485952/s

================================================================================
T=5min S=1KiB VUs=10

--------------------------------------------------------------------------------
master
     data_sent................: 89 MB 295 kB/s
     iteration_duration.......: avg=34.44ms min=2.75µs  med=36.86ms max=51.76ms p(90)=38.56ms p(95)=39.22ms
     iterations...............: 87066 290.18385/s
     local_obj_put_duration...: avg=34.11ms min=10.91ms med=36.55ms max=51.38ms p(90)=38.24ms p(95)=38.88ms
     local_obj_put_total......: 87066 290.18385/s

--------------------------------------------------------------------------------
badger
     data_sent................: 119 MB 396 kB/s
     iteration_duration.......: avg=25.76ms min=1.92µs med=25.87ms max=85.41ms p(90)=27.35ms p(95)=28.01ms
     iterations...............: 116368 387.885353/s
     local_obj_put_duration...: avg=25.46ms min=9.46ms med=25.58ms max=85.05ms p(90)=27.06ms p(95)=27.71ms
     local_obj_put_total......: 116368 387.885353/s

================================================================================
T=5min S=1KiB VUs=100

--------------------------------------------------------------------------------
master
     data_sent................: 393 MB 1.3 MB/s
     iteration_duration.......: avg=77.84ms min=4.16µs  med=79.38ms max=123.36ms p(90)=89.15ms p(95)=92.47ms
     iterations...............: 385245 1284.061976/s
     local_obj_put_duration...: avg=77.4ms  min=17.07ms med=78.97ms max=123.08ms p(90)=88.75ms p(95)=92.05ms
     local_obj_put_total......: 385245 1284.061976/s

--------------------------------------------------------------------------------
badger
     data_sent................: 512 MB 1.7 MB/s
     iteration_duration.......: avg=59.8ms  min=3.37µs med=60.57ms max=247.43ms p(90)=69.97ms p(95)=73.53ms
     iterations...............: 501490 1671.459986/s
     local_obj_put_duration...: avg=59.41ms min=1.75ms med=60.19ms max=247.1ms  p(90)=69.55ms p(95)=73.03ms
     local_obj_put_total......: 501490 1671.459986/s

================================================================================
T=10min S=64KiB VUs=1

--------------------------------------------------------------------------------
master
     data_sent................: 1.2 GB 2.0 MB/s
     iteration_duration.......: avg=32.62ms min=4.54µs  med=33.3ms  max=47.21ms p(90)=34.35ms p(95)=34.99ms
     iterations...............: 18377  30.626653/s
     local_obj_put_duration...: avg=31.97ms min=14.04ms med=32.63ms max=46.41ms p(90)=33.67ms p(95)=34.31ms
     local_obj_put_total......: 18377  30.626653/s

--------------------------------------------------------------------------------
badger
     data_sent................: 1.7 GB 2.8 MB/s
     iteration_duration.......: avg=23.08ms min=4.8µs  med=22.53ms max=249.33ms p(90)=23.07ms p(95)=23.31ms
     iterations...............: 25966  43.275906/s
     local_obj_put_duration...: avg=22.45ms min=2.49ms med=21.9ms  max=248.19ms p(90)=22.42ms p(95)=22.61ms
     local_obj_put_total......: 25966  43.275906/s

================================================================================
T=10min S=64KiB VUs=10

--------------------------------------------------------------------------------
master
     data_sent................: 8.4 GB 14 MB/s
     iteration_duration.......: avg=46.37ms min=3.83µs med=47.7ms  max=750.61ms p(90)=52.94ms p(95)=54.94ms
     iterations...............: 129320 215.51723/s
     local_obj_put_duration...: avg=45.73ms min=16ms   med=47.07ms max=749.93ms p(90)=52.31ms p(95)=54.31ms
     local_obj_put_total......: 129320 215.51723/s

--------------------------------------------------------------------------------
badger
     data_sent................: 10 GB  17 MB/s
     iteration_duration.......: avg=39.06ms min=4.17µs med=26.89ms max=1.11s p(90)=37.65ms p(95)=124.25ms
     iterations...............: 153513 255.826806/s
     local_obj_put_duration...: avg=38.39ms min=2.65ms med=26.27ms max=1.11s p(90)=36.85ms p(95)=123.12ms
     local_obj_put_total......: 153513 255.826806/s

================================================================================
T=10min S=64KiB VUs=100

--------------------------------------------------------------------------------
master
     data_sent................: 23 GB  38 MB/s
     iteration_duration.......: avg=170.55ms min=2.97µs  med=167.11ms max=1.15s p(90)=217.15ms p(95)=231.11ms
     iterations...............: 351803 586.161655/s
     local_obj_put_duration...: avg=169.87ms min=48.03ms med=166.43ms max=1.15s p(90)=216.44ms p(95)=230.45ms
     local_obj_put_total......: 351803 586.161655/s

--------------------------------------------------------------------------------
badger
     data_sent................: 18 GB  31 MB/s
     iteration_duration.......: avg=213.64ms min=8.47µs med=76.76ms max=2s p(90)=593.03ms p(95)=739.56ms
     iterations...............: 280904 467.962847/s
     local_obj_put_duration...: avg=212.7ms  min=2.03ms med=75.97ms max=2s p(90)=591.85ms p(95)=737.94ms
     local_obj_put_total......: 280904 467.962847/s
I reran the tests using the correct volumes: ``` ================================================================================ T=1min S=8KiB VUs=1 -------------------------------------------------------------------------------- master data_sent................: 16 MB 258 kB/s iteration_duration.......: avg=31.7ms min=3.86µs med=32.25ms max=47.26ms p(90)=32.89ms p(95)=33.05ms iterations...............: 1890 31.490094/s local_obj_put_duration...: avg=31.33ms min=19.24ms med=31.84ms max=46.42ms p(90)=32.47ms p(95)=32.63ms local_obj_put_total......: 1890 31.490094/s -------------------------------------------------------------------------------- badger data_sent................: 23 MB 374 kB/s iteration_duration.......: avg=21.85ms min=3.31µs med=21.75ms max=43.12ms p(90)=22.46ms p(95)=22.57ms iterations...............: 2741 45.677653/s local_obj_put_duration...: avg=21.49ms min=17.19ms med=21.39ms max=42.82ms p(90)=22.04ms p(95)=22.16ms local_obj_put_total......: 2741 45.677653/s ================================================================================ T=1min S=8KiB VUs=10 -------------------------------------------------------------------------------- master data_sent................: 138 MB 2.3 MB/s iteration_duration.......: avg=35.4ms min=4.46µs med=37.16ms max=52.91ms p(90)=38.81ms p(95)=39.6ms iterations...............: 16940 282.264017/s local_obj_put_duration...: avg=35.04ms min=16.32ms med=36.81ms max=52.64ms p(90)=38.47ms p(95)=39.23ms local_obj_put_total......: 16940 282.264017/s -------------------------------------------------------------------------------- badger data_sent................: 193 MB 3.2 MB/s iteration_duration.......: avg=25.29ms min=3.49µs med=25.18ms max=113.74ms p(90)=26.61ms p(95)=27.26ms iterations...............: 23705 395.047146/s local_obj_put_duration...: avg=24.95ms min=15.85ms med=24.85ms max=113.38ms p(90)=26.28ms p(95)=26.91ms local_obj_put_total......: 23705 395.047146/s ================================================================================ T=1min S=8KiB VUs=100 -------------------------------------------------------------------------------- master data_sent................: 611 MB 10 MB/s iteration_duration.......: avg=80.23ms min=3.85µs med=81.47ms max=121.99ms p(90)=92.87ms p(95)=97ms iterations...............: 74809 1245.80559/s local_obj_put_duration...: avg=79.76ms min=32.5ms med=81.02ms max=121.59ms p(90)=92.45ms p(95)=96.57ms local_obj_put_total......: 74809 1245.80559/s -------------------------------------------------------------------------------- badger data_sent................: 873 MB 15 MB/s iteration_duration.......: avg=56.17ms min=2.25µs med=54.23ms max=302.9ms p(90)=64.14ms p(95)=74.92ms iterations...............: 106811 1779.295693/s local_obj_put_duration...: avg=55.81ms min=2.07ms med=53.87ms max=302.51ms p(90)=63.77ms p(95)=74.52ms local_obj_put_total......: 106811 1779.295693/s ================================================================================ T=5min S=1KiB VUs=1 -------------------------------------------------------------------------------- master data_sent................: 9.7 MB 32 kB/s iteration_duration.......: avg=31.53ms min=4.4µs med=32.16ms max=44.83ms p(90)=32.92ms p(95)=33.08ms iterations...............: 9504 31.679499/s local_obj_put_duration...: avg=31.17ms min=15.69ms med=31.78ms max=43.66ms p(90)=32.54ms p(95)=32.7ms local_obj_put_total......: 9504 31.679499/s -------------------------------------------------------------------------------- badger data_sent................: 14 MB 47 kB/s iteration_duration.......: avg=21.96ms min=2.59µs med=21.93ms max=32.09ms p(90)=22.51ms p(95)=22.64ms iterations...............: 13646 45.485952/s local_obj_put_duration...: avg=21.62ms min=15.29ms med=21.6ms max=31.51ms p(90)=22.15ms p(95)=22.27ms local_obj_put_total......: 13646 45.485952/s ================================================================================ T=5min S=1KiB VUs=10 -------------------------------------------------------------------------------- master data_sent................: 89 MB 295 kB/s iteration_duration.......: avg=34.44ms min=2.75µs med=36.86ms max=51.76ms p(90)=38.56ms p(95)=39.22ms iterations...............: 87066 290.18385/s local_obj_put_duration...: avg=34.11ms min=10.91ms med=36.55ms max=51.38ms p(90)=38.24ms p(95)=38.88ms local_obj_put_total......: 87066 290.18385/s -------------------------------------------------------------------------------- badger data_sent................: 119 MB 396 kB/s iteration_duration.......: avg=25.76ms min=1.92µs med=25.87ms max=85.41ms p(90)=27.35ms p(95)=28.01ms iterations...............: 116368 387.885353/s local_obj_put_duration...: avg=25.46ms min=9.46ms med=25.58ms max=85.05ms p(90)=27.06ms p(95)=27.71ms local_obj_put_total......: 116368 387.885353/s ================================================================================ T=5min S=1KiB VUs=100 -------------------------------------------------------------------------------- master data_sent................: 393 MB 1.3 MB/s iteration_duration.......: avg=77.84ms min=4.16µs med=79.38ms max=123.36ms p(90)=89.15ms p(95)=92.47ms iterations...............: 385245 1284.061976/s local_obj_put_duration...: avg=77.4ms min=17.07ms med=78.97ms max=123.08ms p(90)=88.75ms p(95)=92.05ms local_obj_put_total......: 385245 1284.061976/s -------------------------------------------------------------------------------- badger data_sent................: 512 MB 1.7 MB/s iteration_duration.......: avg=59.8ms min=3.37µs med=60.57ms max=247.43ms p(90)=69.97ms p(95)=73.53ms iterations...............: 501490 1671.459986/s local_obj_put_duration...: avg=59.41ms min=1.75ms med=60.19ms max=247.1ms p(90)=69.55ms p(95)=73.03ms local_obj_put_total......: 501490 1671.459986/s ================================================================================ T=10min S=64KiB VUs=1 -------------------------------------------------------------------------------- master data_sent................: 1.2 GB 2.0 MB/s iteration_duration.......: avg=32.62ms min=4.54µs med=33.3ms max=47.21ms p(90)=34.35ms p(95)=34.99ms iterations...............: 18377 30.626653/s local_obj_put_duration...: avg=31.97ms min=14.04ms med=32.63ms max=46.41ms p(90)=33.67ms p(95)=34.31ms local_obj_put_total......: 18377 30.626653/s -------------------------------------------------------------------------------- badger data_sent................: 1.7 GB 2.8 MB/s iteration_duration.......: avg=23.08ms min=4.8µs med=22.53ms max=249.33ms p(90)=23.07ms p(95)=23.31ms iterations...............: 25966 43.275906/s local_obj_put_duration...: avg=22.45ms min=2.49ms med=21.9ms max=248.19ms p(90)=22.42ms p(95)=22.61ms local_obj_put_total......: 25966 43.275906/s ================================================================================ T=10min S=64KiB VUs=10 -------------------------------------------------------------------------------- master data_sent................: 8.4 GB 14 MB/s iteration_duration.......: avg=46.37ms min=3.83µs med=47.7ms max=750.61ms p(90)=52.94ms p(95)=54.94ms iterations...............: 129320 215.51723/s local_obj_put_duration...: avg=45.73ms min=16ms med=47.07ms max=749.93ms p(90)=52.31ms p(95)=54.31ms local_obj_put_total......: 129320 215.51723/s -------------------------------------------------------------------------------- badger data_sent................: 10 GB 17 MB/s iteration_duration.......: avg=39.06ms min=4.17µs med=26.89ms max=1.11s p(90)=37.65ms p(95)=124.25ms iterations...............: 153513 255.826806/s local_obj_put_duration...: avg=38.39ms min=2.65ms med=26.27ms max=1.11s p(90)=36.85ms p(95)=123.12ms local_obj_put_total......: 153513 255.826806/s ================================================================================ T=10min S=64KiB VUs=100 -------------------------------------------------------------------------------- master data_sent................: 23 GB 38 MB/s iteration_duration.......: avg=170.55ms min=2.97µs med=167.11ms max=1.15s p(90)=217.15ms p(95)=231.11ms iterations...............: 351803 586.161655/s local_obj_put_duration...: avg=169.87ms min=48.03ms med=166.43ms max=1.15s p(90)=216.44ms p(95)=230.45ms local_obj_put_total......: 351803 586.161655/s -------------------------------------------------------------------------------- badger data_sent................: 18 GB 31 MB/s iteration_duration.......: avg=213.64ms min=8.47µs med=76.76ms max=2s p(90)=593.03ms p(95)=739.56ms iterations...............: 280904 467.962847/s local_obj_put_duration...: avg=212.7ms min=2.03ms med=75.97ms max=2s p(90)=591.85ms p(95)=737.94ms local_obj_put_total......: 280904 467.962847/s ```
ale64bit commented 2023-07-03 06:36:28 +00:00
Member
Copy link

Badger seems to be worse for the 64KiB case.

One of the downsides of rewriting writecache this way, rather than just replacing the underlying store as in #454, is that now it's harder to gauge what is the concrete cause of improvements or regressions (it could be badger itself, but the key encoding, removing fstree and so on, could also have an effect).

> Badger seems to be worse for the 64KiB case. One of the downsides of rewriting writecache this way, rather than just replacing the underlying store as in https://git.frostfs.info/TrueCloudLab/frostfs-node/pulls/454, is that now it's harder to gauge what is the concrete cause of improvements or regressions (it could be badger itself, but the key encoding, removing fstree and so on, could also have an effect).
dstepanov-yadro commented 2023-07-03 07:02:25 +00:00
Member
Copy link
T=10min S=64KiB VUs=100

--------------------------------------------------------------------------------
master
     data_sent................: 23 GB  38 MB/s
badger
     data_sent................: 18 GB  31 MB/s

Were metrics collected during tests? It is unclear if this bbolt degradation is permanent, or was there some kind of stop the world?

``` T=10min S=64KiB VUs=100 -------------------------------------------------------------------------------- master data_sent................: 23 GB 38 MB/s badger data_sent................: 18 GB 31 MB/s ``` Were metrics collected during tests? It is unclear if this bbolt degradation is permanent, or was there some kind of stop the world?
ale64bit commented 2023-07-03 07:15:41 +00:00
Member
Copy link

(I assume you mean badger degradation)
I didn't collect the metrics unfortunately, but I would also like to collect some data with time information. I don't think badger has a stop-the-world GC strategy, but still (it runs GC every minute).

(I assume you mean badger degradation) I didn't collect the metrics unfortunately, but I would also like to collect some data with time information. I don't think badger has a stop-the-world GC strategy, but still (it runs GC every minute).
fyrchik commented 2024-01-16 12:02:02 +00:00
Author
Owner
Copy link

We tried, but the work wasn't finished and the support cost was nonnegligible. So it was dropped in #887
Bitcask implementation seems like a better choice (refs #654), it also doesn't suffer from all "panic when disk is removed" problems, because it doesn't use mmap.

We tried, but the work wasn't finished and the support cost was nonnegligible. So it was dropped in #887 Bitcask implementation seems like a better choice (refs #654), it also doesn't suffer from all "panic when disk is removed" problems, because it doesn't use mmap.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
support/v0.42
support/v0.38
support/v0.37
support/v0.36
support/v0.34
support/v0.30
support/v0.27
v0.44.0-rc.13
v0.44.0-rc.12
v0.44.0-rc.11
v0.42.16
v0.44.0-rc.10
v0.44.0-rc.9
v0.44.0-rc.8
v0.44.0-rc.7
v0.44.0-rc.6
v0.44.0-rc.5
v0.44.0-rc.4
v0.44.0-rc.3
v0.44.0-rc.2
v0.44.0-rc.1
v0.43.2
v0.43.1
v0.43.0
v0.42.15
v0.42.14
v0.42.13
v0.42.12
v0.42.11
v0.42.10
v0.42.9
v0.42.8
v0.42.7
v0.42.6
v0.42.5
v0.42.4
v0.42.3
v0.42.2
v0.42.1
v0.42.0
v0.42.0-rc.9
v0.42.0-rc.8
v0.42.0-rc.7
v0.38.9
v0.42.0-rc.6
v0.42.0-rc.5
v0.42.0-rc.4
v0.42.0-rc.3
v0.42.0-rc.2
v0.42.0-rc.1
v0.38.8
v0.41.0
v0.38.7
v0.40.0
v0.39.0
v0.38.6
v0.38.5
v0.38.4
v0.38.3
v0.38.2
v0.38.1
v0.38.0
v0.38.0-rc.2
v0.38.0-rc.1
v0.37.0
v0.37.0-rc.1
v0.36.0
v0.34.0
v0.22.1
v0.22.0
v0.21.1
v0.21.0
v0.20.0
v0.19.0
v0.18.0
v0.17.0
v0.16.0
v0.15.0
v0.14.3
v0.14.2
v0.14.1
v0.14.0
v0.14.0-rc.1
v0.13.2
v0.13.1
v0.13.0
v0.13.0-rc.1
v0.12.1
v0.12.0
v0.12.0-rc3
v0.12.0-rc2
v0.12.0-rc1
v0.11.0
v0.10.0
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
badger

Badger DB related issues

  
frostfs-adm

   
frostfs-cli

   
frostfs-ir

   
frostfs-lens

   
frostfs-node

   
good first issue

Good for newcomers

  
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
vNext
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
ale64bit
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#421
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?