Improve batching in badger #636

New issue

Closed

opened 2023-08-22 08:30:42 +00:00 by fyrchik · 1 comment

fyrchik commented

2023-08-22 08:30:42 +00:00

Owner

Batch() in bolt accumulates tx from different goroutines implicitly, so they are grouped and synced together.
AFAIK badger does no such thing, so we may want add grouping in our own code.

My suggestion:

Writecache has a global Batch instance and flush channel.
After some time or elements accumulated batch is flushed.
Caller receives an error from some flushCh belonging to a batch (all batches receive the same error, but make sure to handle ErrTxnTooBig on the caller -- we may fallback to a simple Set in this case.

Related #627.

`Batch()` in bolt accumulates tx from different goroutines implicitly, so they are grouped and synced together. AFAIK badger does no such thing, so we may want add grouping in our own code. My suggestion: 1. Writecache has a global `Batch` instance and flush channel. 2. After some time or elements accumulated batch is flushed. 3. Caller receives an error from some `flushCh` belonging to a batch (all batches receive the same error, but make sure to handle `ErrTxnTooBig` on the caller -- we may fallback to a simple `Set` in this case. Related #627.