TrueCloudLab/frostfs-node

Fork 27

Add new method for listing containers via stream #1452

New issue

Closed

opened 2024-10-28 15:13:57 +00:00 by elebedeva · 1 comment

elebedeva commented

2024-10-28 15:13:57 +00:00

Member

There is a problem with container list command when the number of containers exceeds a certain amount (about 130k).

$ frostfs-cli -c work-cfg.yml container list
rpc error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4894749 vs. 4194304)

Describe the solution you'd like

Add new method returning stream for listing containers.

Changes are expected in repos:

frostfs-api
frostfs-api-go
frostfs-sdk-go
frostfs-node

## Is your feature request related to a problem? Please describe. There is a problem with `container list` command when the number of containers exceeds a certain amount (about 130k). ``` $ frostfs-cli -c work-cfg.yml container list rpc error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4894749 vs. 4194304) ``` ## Describe the solution you'd like Add new method returning stream for listing containers. Changes are expected in repos: - [x] frostfs-api - [x] frostfs-api-go - [x] frostfs-sdk-go - [x] frostfs-node

elebedeva added the

triage

label 2024-10-28 15:13:57 +00:00

elebedeva referenced this issue from TrueCloudLab/frostfs-api

2024-10-28 15:16:41 +00:00

container: Add ListStream method #70

elebedeva referenced this issue from TrueCloudLab/frostfs-api-go

2024-10-28 15:20:04 +00:00

WIP: container: Add ListStream method #125

elebedeva referenced this issue from TrueCloudLab/frostfs-sdk-go

2024-10-28 15:21:28 +00:00

container: Add ListStream method #291

elebedeva referenced this issue

2024-10-28 15:22:41 +00:00

container: Add ListStream method #1453

fyrchik added the

frostfs-node

label 2024-10-28 15:51:58 +00:00

fyrchik added this to the v0.44.0 milestone 2024-10-28 15:52:02 +00:00

elebedeva was assigned by fyrchik

2024-10-28 15:52:08 +00:00

fyrchik removed the

triage

label 2024-10-28 15:52:12 +00:00

fyrchik modified the milestone from v0.44.0 to vNext

2024-10-28 15:52:22 +00:00

fyrchik referenced this issue from a commit

2024-12-17 13:31:35 +00:00

[#1452] container: Add ListStream method

elebedeva commented

2024-12-27 11:59:11 +00:00

Author

Member

Next step is to combine iteration through a container identifiers in morph and in container service when sending identifiers via stream.

First implementation reused existing morph functions for reading cids from a contract. So we at first read all cids to a list and then send them.

It would be much more efficient to send batches of cids when reading them from the contract. It would reduce amount of iterations through ciontainer list.

Next step is to combine iteration through a container identifiers in `morph` and in `container service` when sending identifiers via `stream`. First implementation reused existing `morph` functions for reading `cid`s from a contract. So we _at first_ read all `cid`s to a list and _then_ send them. It would be much more efficient to send batches of `cid`s _when_ reading them from the contract. It would reduce amount of iterations through `ciontainer list`.

elebedeva referenced this issue from a pull request that will close it,

2024-12-27 12:00:13 +00:00

container: Reduce iterations through container list #1577

fyrchik modified the milestone from vNext to v0.45.0

2024-12-27 12:09:23 +00:00

fyrchik closed this issue

2024-12-28 10:05:37 +00:00