Support object splitting in the pool #114
Labels
No labels
P0
P1
P2
P3
good first issue
pool
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No milestone
No project
No assignees
1 participant
dkirillov
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-sdk-go#114
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Is your feature request related to a problem? Please describe.
Pool's object put method process streaming data: pool pipes the stream into storage node. While this makes pool component lightweight, pool can't use retries and produce bigger network load.
As an alternative, pool can split and prepare object on the client side, buffer it and send back to the node. In this case pool can retry request or choose more suitable node to send the request.
Describe the solution you'd like
Provide new method in object pool interface to upload objects. The signature should be similar to a existing method, so the client's won't need to adapt code to it.
Implementation should split and prepare object before sending the request. To do that it should allocate a memory for max object payload in the network.
Note! Pool should control amount of memory it allocates. It should have hard limit of some sort and consider blocking / non-blocking modes for object put invocations.
Describe alternatives you've considered
Organize object split on the client side and provide interface to upload split objects. This code is going to be boilerplate so it is more appropriate to define it in the pool.
Additional context