Improve multipart removal speed #559

New issue

Closed

opened 2024-11-21 08:09:24 +00:00 by alexvanin · 1 comment

alexvanin commented

2024-11-21 08:09:24 +00:00

Owner

After #63 multipart object is a list of multiple native FrostFS objects. To delete such object, gateway sends object.Delete request on every part consecutively. This is not very efficient for large multipart objects, which may consist up to 10 000 parts and most clients run out of time.

Describe the solution you'd like

To improve removal efficiency there are some new changes.

Replace object.Delete operation with object.Put operation. Tombstone object may consist multiple objects, so it is more efficient to group them and not having separate tombstone for every object part. However there is still trade-off between tombstone size and object.Put duration. It will improve over time with frostfs-node#1450, though. So we have to decide on what tombstone size is suitable for different environments.
Send tombstones in pool of go routines.

These two changes should improve removal speed.

Describe alternatives you've considered

Postponed removal. This might be an evolution of proposal: create an async queue and store here all parts to remove. This may be done by external services such as lifecycler service.

## Is your feature request related to a problem? Please describe. After #63 multipart object is a list of multiple native FrostFS objects. To delete such object, gateway sends `object.Delete` request on every part consecutively. This is not very efficient for large multipart objects, which may consist up to 10 000 parts and most clients run out of time. ## Describe the solution you'd like To improve removal efficiency there are some new changes. 1. Replace `object.Delete` operation with `object.Put` operation. Tombstone object may consist multiple objects, so it is more efficient to group them and not having separate tombstone for every object part. **However** there is still trade-off between tombstone size and `object.Put` duration. It will improve over time with [frostfs-node#1450](https://git.frostfs.info/TrueCloudLab/frostfs-node/issues/1450), though. So we have to decide on what tombstone size is suitable for different environments. 2. Send tombstones in pool of go routines. These two changes should improve removal speed. ## Describe alternatives you've considered Postponed removal. This might be an evolution of proposal: create an async queue and store here all parts to remove. This may be done by external services such as lifecycler service.

mbiryukova was assigned by alexvanin

2024-11-21 08:09:24 +00:00

alexvanin added this to the v0.32.0 milestone 2024-11-21 08:10:11 +00:00

fyrchik commented

2024-11-21 12:03:36 +00:00

Owner

That said, we still might hit node timeouts.

Regarding your first point, doing tombstone PUT looks a lot better to me, as it doesn't create session and may lead to a simpler APE chain set.

Just to be clear: packing everything in one tombstone is the fastest solution, because we still eventually need to remove every ID in the list. So it is 1 30-second requests vs 30 1-second requests. Experimental data obtained by @a-savchuk supports this https://git.frostfs.info/TrueCloudLab/frostfs-node/issues/1450#issuecomment-56634 That said, we still might hit node timeouts. Regarding your first point, doing tombstone PUT looks a lot better to me, as it doesn't create session and may lead to a simpler APE chain set.

👍 1

mbiryukova referenced this issue from a pull request that will close it,

2024-11-22 09:51:46 +00:00

[#559] Remove multipart objects using tombstones #560

alexvanin referenced this issue from a commit

2024-12-04 08:16:13 +00:00

[#559] Remove multipart objects using tombstones

alexvanin closed this issue

2024-12-04 08:16:13 +00:00

mbiryukova referenced this issue

2024-12-11 04:22:41 +00:00

[#559] Return error from multipart deleting #574