Support object splitting in the pool #114
Labels
No Label
P0
P1
P2
P3
good first issue
pool
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No Milestone
No Assignees
1 Participants
dkirillov
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-sdk-go#114
Loading…
Reference in New Issue
There is no content yet.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?
Is your feature request related to a problem? Please describe.
Pool's object put method process streaming data: pool pipes the stream into storage node. While this makes pool component lightweight, pool can't use retries and produce bigger network load.
As an alternative, pool can split and prepare object on the client side, buffer it and send back to the node. In this case pool can retry request or choose more suitable node to send the request.
Describe the solution you'd like
Provide new method in object pool interface to upload objects. The signature should be similar to a existing method, so the client's won't need to adapt code to it.
Implementation should split and prepare object before sending the request. To do that it should allocate a memory for max object payload in the network.
Note! Pool should control amount of memory it allocates. It should have hard limit of some sort and consider blocking / non-blocking modes for object put invocations.
Describe alternatives you've considered
Organize object split on the client side and provide interface to upload split objects. This code is going to be boilerplate so it is more appropriate to define it in the pool.
Additional context