Parallelize object distribution over all the nodes #225

New issue

Open

opened 2023-04-06 17:04:18 +00:00 by carpawell · 1 comment

carpawell commented

2023-04-06 17:04:18 +00:00

Contributor

Currently Next can handle placement vectors one by one only. Every vector is being handled asyncly but I think it is possible to work with every node (in every vector and not only in the same one) that way too. But it needs to be done carefully without Put requests duplication.

Currently [`Next`](https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/90276953719d4cf74a94a2fc10d98f1fac0aa4d9/pkg/services/object_manager/placement/traverser.go#L164-L188) can handle placement vectors one by one only. Every vector is being handled [asyncly](https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/90276953719d4cf74a94a2fc10d98f1fac0aa4d9/pkg/services/object/put/distributed.go#L215-L256) but I think it is possible to work with every node (in every vector and not only in the same one) that way too. But it needs to be done carefully without `Put` requests duplication.

carpawell added the

triage

label 2023-04-06 17:04:18 +00:00

carpawell referenced this issue

2023-04-06 17:13:54 +00:00

Do our best to prevent double replication while Put handling #226

fyrchik added the

frostfs-node

label 2023-04-07 04:57:00 +00:00

snegurochka added the

enhancement

label 2023-05-03 17:14:44 +00:00

fyrchik added this to the vNext milestone 2023-05-18 08:32:32 +00:00

fyrchik referenced this issue

2023-08-24 08:07:49 +00:00

Refactor traversing container nodes in services #643

fyrchik added

and removed

triage

labels 2024-01-16 12:11:09 +00:00

fyrchik commented

2024-01-16 12:13:00 +00:00

Owner

Multiple REP vectors is not the most popular scenario, and the implementation needs to play nicely with copies_number parameter -- we must store first N copies according to the order.
So either we use this optimization when copies_number=0 or do not use at all.

Multiple REP vectors is not the most popular scenario, and the implementation needs to play nicely with `copies_number` parameter -- we must store first N copies according to the order. So either we use this optimization when `copies_number=0` or do not use at all.

fyrchik added the

discussion

triage

labels 2024-01-16 12:13:17 +00:00