Do not save objects to metabase on PUT if writecache enabled #992

New issue

Open

opened 2024-02-16 14:03:46 +00:00 by dstepanov-yadro · 4 comments

dstepanov-yadro commented

2024-02-16 14:03:46 +00:00

Member

How PUT works now (if writecache enabled):

object saved to writecache
object saved to metabase
object flushed from writecache to blobstore

Performance tests on hardware show that latency of saving to writecache and to metabase are almost the same. So for pretty small objects (8KB for example) total PUT latency = save to writecache latency + save to metabase latency.

Describe the solution you'd like

Writecache should be fast enough to read whole object. So maybe it is possible to change PUT algorithm:

object saved to writecache
object flushed from writecache to metabase and blobstore

I suppose to investigate the possibility of changing the PUT behavior as described, and if possible to implement it.

## Is your feature request related to a problem? Please describe. How PUT works now (if writecache enabled): - object saved to writecache - object saved to metabase - object flushed from writecache to blobstore Performance tests on hardware show that latency of saving to writecache and to metabase are almost the same. So for pretty small objects (8KB for example) total PUT latency = save to writecache latency + save to metabase latency. ## Describe the solution you'd like Writecache should be fast enough to read whole object. So maybe it is possible to change PUT algorithm: - object saved to writecache - object flushed from writecache to metabase and blobstore I suppose to investigate the possibility of changing the PUT behavior as described, and if possible to implement it.

dstepanov-yadro added the

labels 2024-02-16 14:03:46 +00:00

dstepanov-yadro commented

2024-02-16 14:16:22 +00:00

Author

Member

Important cases:

object in writecache locked and lock expired
object in writecache inhumed by policer or by user

Important cases: - object in writecache locked and lock expired - object in writecache inhumed by policer or by user

fyrchik commented

2024-02-16 14:30:57 +00:00

Owner

This will complicate GET operation: we can no longer rely on the metabase to see whether an object exists and this must be taken into account everywhere inside the shard. On the engine level we can rely on the shard interface.
This could also complicate iteration: we have objects in the writecache and objects in the metabase, and some of them could appear twice. "First Metabase Then Writecache" will work, but has side-effects: object A is put after B, but we see A, but not B during iteration (because B is being flushed, while A is in the writecache).

About expired/inhumed objects: can such objects be put by user? The only case I can think of is malicious actor or lagging storage node. In both cases "success" status shouldn't hurt anything.
Locking is different though, the returned status is important.

1. This will complicate GET operation: we can no longer rely on the metabase to see whether an object exists and this must be taken into account everywhere _inside_ the shard. On the engine level we can rely on the shard interface. 2. This could also complicate iteration: we have objects in the writecache and objects in the metabase, and some of them could appear twice. "First Metabase Then Writecache" will work, but has side-effects: object A is put after B, but we see A, but not B during iteration (because B is being flushed, while A is in the writecache). About expired/inhumed objects: can such objects be put by user? The only case I can think of is malicious actor or lagging storage node. In both cases "success" status shouldn't hurt anything. Locking is different though, the returned status is important.

fyrchik commented

2024-02-16 14:33:28 +00:00

Owner

Crazy, but what if small objects are put directly to the metabase (i.e. no CutPayload)?
And flushing is done from the metabase by some background thread.

Crazy, but what if small objects are put directly to the metabase (i.e. no `CutPayload`)? And flushing is done from the metabase by some background thread.

dstepanov-yadro commented

2024-02-16 14:44:04 +00:00

Author

Member