Investigate perfomance for bigger gRPC message sizes #77

New issue

Closed

opened 2023-02-28 11:49:13 +00:00 by fyrchik · 2 comments

fyrchik commented

2023-02-28 11:49:13 +00:00

(Migrated from github.com)

By default it is 4 MiB. This leads to big objects being split into 4MiB chunks, thus for MaxObjectSize=64 MiB we have at least 16 messages sent (don't forget about signing and verification). For custom deployments where we have full control over the network we could set this size depending on MaxObjectSize both on client and server.

In this task:

Set grpc.MaxRecvMsgSize in node to some high value (70 MiB).
Perform benchmarks of 64 MiB objects with custom client build (see https://github.com/nspcc-dev/neofs-node/pull/2243 for an example on what needs to be changed).
If observations support the hypothesis, add a config parameter for this and create tasks to support this on client:

api-go and sdk-go
k6
frostfs-cli parameter

In theory this enables future optimizations, such as being able to replicate an object from the blobstor without unmarshaling. (in this case, also check that validation is done when the object is received, we don't want to propagate possibly corrupted data across the cluster, see https://www.usenix.org/system/files/conference/fast17/fast17-ganesan.pdf).

By default it is 4 MiB. This leads to big objects being split into 4MiB chunks, thus for MaxObjectSize=64 MiB we have _at least_ 16 messages sent (don't forget about signing and verification). For custom deployments where we have full control over the network we could set this size depending on MaxObjectSize both on client and server. In this task: 1. Set `grpc.MaxRecvMsgSize` in node to some high value (70 MiB). 2. Perform benchmarks of 64 MiB objects with custom client build (see https://github.com/nspcc-dev/neofs-node/pull/2243 for an example on what needs to be changed). 3. If observations support the hypothesis, add a config parameter for this and create tasks to support this on client: - api-go and sdk-go - k6 - frostfs-cli parameter In theory this enables future optimizations, such as being able to replicate an object from the blobstor without unmarshaling. (in this case, also check that validation is done when the object is received, we don't want to propagate _possibly corrupted_ data across the cluster, see https://www.usenix.org/system/files/conference/fast17/fast17-ganesan.pdf). Somewhat related https://github.com/TrueCloudLab/frostfs-api/issues/9

👍 1

fyrchik added the

frostfs-node

label 2023-05-04 06:45:22 +00:00

fyrchik added this to the v0.38.0 milestone 2023-05-18 08:49:17 +00:00

dstepanov-yadro self-assigned this 2023-06-22 12:13:55 +00:00

dstepanov-yadro commented

2023-06-26 15:16:59 +00:00

Member

What did I do:

set frostfs-node limit on the maximum gRPC message size of 2GB
built frostfs-node with frostfs-sdk-go version, in which I changed the size of the chunk to 5 MB
ran the preset into one stream with REP 1 SELECT 1 with the load size from the previous point
repeated for several chunk sizes until the download speed stopped growing

It turned out that on my PC the optimal chunk size is 60MB (+/-). That is, the speed no longer increases, but even drops a little.
If we compare the download speed of an object of 60MB with a chunk of 60MB and with a chunk of 3 MB, we get 36 MB/s and 27 MB/s, respectively.

I think we can do the following:

frostfs-node: remove the maximum size of the received message and remove the maximum size of the sent message
frostfs-sdk-go: add my changes to set the chunk size (TrueCloudLab/frostfs-sdk-go#96)
frostfs-cli: add a parameter to set the chunk size
test on hardware

What did I do: - set frostfs-node limit on the maximum gRPC message size of 2GB - built frostfs-node with frostfs-sdk-go version, in which I changed the size of the chunk to 5 MB - ran the preset into one stream with ```REP 1 SELECT 1``` with the load size from the previous point - repeated for several chunk sizes until the download speed stopped growing It turned out that on my PC the optimal chunk size is 60MB (+/-). That is, the speed no longer increases, but even drops a little. If we compare the download speed of an object of 60MB with a chunk of 60MB and with a chunk of 3 MB, we get 36 MB/s and 27 MB/s, respectively. I think we can do the following: - frostfs-node: remove the maximum size of the received message and remove the maximum size of the sent message - frostfs-sdk-go: add my changes to set the chunk size (https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pulls/96) - frostfs-cli: add a parameter to set the chunk size - test on hardware

dstepanov-yadro referenced this issue

2023-06-26 15:33:23 +00:00

grpc: Increase message limits #470

dstepanov-yadro referenced this issue from a pull request that will close it,

2023-07-05 09:38:14 +00:00

Add chunk size flag to frostfs-cli object put #491

dstepanov-yadro referenced this issue from TrueCloudLab/xk6-frostfs

2023-07-05 12:36:44 +00:00

Allow to pass custom gRPC chunk size for object put #79

dstepanov-yadro commented

2023-07-06 14:24:15 +00:00

Member