Improve flushing scheme for badger-backed writecache #568

New issue

Closed

opened 2023-08-07 07:32:26 +00:00 by ale64bit · 2 comments

ale64bit commented

2023-08-07 07:32:26 +00:00

Member

The batching scheme used during boltdb-backed writecache implementation might not be relevant to a badger-backed implementation. Consider improving.

See: #462 (comment)
See:

// We put objects in batches of fixed size to not interfere with main put cycle a lot.

The batching scheme used during boltdb-backed writecache implementation might not be relevant to a badger-backed implementation. Consider improving. See: https://git.frostfs.info/TrueCloudLab/frostfs-node/pulls/462#issuecomment-17572 See: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/65c72f3e0b0a2095625f6f7b278645bec7173199/pkg/local_object_storage/writecache/flush.go#L98

ale64bit added the

enhancement

frostfs-node

triage

labels 2023-08-07 07:32:26 +00:00

ale64bit referenced this issue

2023-08-07 07:32:44 +00:00

[#421] Try using badger for the write-cache #462

acid-ant was assigned by fyrchik

2023-08-08 11:11:50 +00:00

fyrchik added the

badger

label 2023-08-09 10:51:11 +00:00

fyrchik commented

2023-08-09 10:56:05 +00:00

Owner

To flush we iterate over everything, but badger also has db.Subscribe which can come handy

To flush we iterate over everything, but badger also has `db.Subscribe` which can come handy

acid-ant commented

2023-08-15 08:59:08 +00:00

Member

To flush we iterate over everything, but badger also has db.Subscribe which can come handy

db.Subscribe is blocking call, so we need to run it in a separate routine. But in this case there are possible situation when we lost some changes. See this PR for details: https://github.com/dgraph-io/badger/pull/1834
It looks like much better to use badger.Stream

> To flush we iterate over everything, but badger also has `db.Subscribe` which can come handy `db.Subscribe` is blocking call, so we need to run it in a separate routine. But in this case there are possible situation when we lost some changes. See this PR for details: https://github.com/dgraph-io/badger/pull/1834 It looks like much better to use [badger.Stream](https://dgraph.io/docs/badger/get-started/#stream)

fyrchik added this to the v0.38.0 milestone 2023-08-23 10:41:35 +00:00

acid-ant referenced this issue from a pull request that will close it,

2023-08-23 11:53:50 +00:00

writecache: Improve flushing scheme for badger #641

fyrchik closed this issue

2023-08-30 17:22:33 +00:00

fyrchik referenced this issue from a commit

2023-08-30 17:22:34 +00:00

[#568] writecache: Improve flushing scheme for badger