2 times degraded PUT rate and PUT payload after 5 hours load of small objects #95

New issue

Closed

opened 2023-03-08 23:30:49 +00:00 by snegurochka · 2 comments

snegurochka commented

2023-03-08 23:30:49 +00:00

Member

Original issue: https://github.com/nspcc-dev/neofs-node/issues/2253

Expected Behavior

We expect linear graph of PUT and GET.

Current Behavior

2 times degraded PUT rate and PUT payload after 5 hours load of small objects.
Linear graph of GET.

Steps to Reproduce (for bugs)

k6 load of small objects on 6h:

k6 run -e DURATION=21600 -e WRITERS=40 -e READERS=40 -e DELETERS=0 -e DELETE_AGE=10 -e REGISTRY_FILE=/var/data/tmp/grpc_reg_090223-nightly-1.json -e WRITE_OBJ_SIZE=120 -e GRPC_ENDPOINTS=10.78.70.114:8080,10.78.70.115:8080,10.78.70.116:8080,10.78.70.117:8080 -e PREGEN_JSON=/var/data/tmp/grpc_090223-nightly-1.json -e STREAM_TIMEOUT=30 -e SLEEP_READ=0.2 -e SLEEP_WRITE=0.2 /home/service/xk6-frostfs/scenarios/grpc.js

Context

Longivity load.

Version

NeoFS Storage node
Version: v0.35.0-41-g5bdf8048
GoVersion: go1.19.5

Your Environment

Server setup and configuration:
HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 https://github.com/nspcc-dev/neofs-node/issues/1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

Original issue: https://github.com/nspcc-dev/neofs-node/issues/2253 ## Expected Behavior We expect linear graph of PUT and GET. ## Current Behavior 2 times degraded PUT rate and PUT payload after 5 hours load of small objects. Linear graph of GET. ![image](https://user-images.githubusercontent.com/110966367/218111885-c4c1a441-75f6-49ab-b647-3dcb46ffd842.png) ![image](https://user-images.githubusercontent.com/110966367/218111982-6fbe8812-e2d5-4eda-ba02-4b4e7bf560e9.png) ## Steps to Reproduce (for bugs) k6 load of small objects on 6h: ``` k6 run -e DURATION=21600 -e WRITERS=40 -e READERS=40 -e DELETERS=0 -e DELETE_AGE=10 -e REGISTRY_FILE=/var/data/tmp/grpc_reg_090223-nightly-1.json -e WRITE_OBJ_SIZE=120 -e GRPC_ENDPOINTS=10.78.70.114:8080,10.78.70.115:8080,10.78.70.116:8080,10.78.70.117:8080 -e PREGEN_JSON=/var/data/tmp/grpc_090223-nightly-1.json -e STREAM_TIMEOUT=30 -e SLEEP_READ=0.2 -e SLEEP_WRITE=0.2 /home/service/xk6-frostfs/scenarios/grpc.js ``` ## Context Longivity load. ## Version ``` NeoFS Storage node Version: v0.35.0-41-g5bdf8048 GoVersion: go1.19.5 ``` ## Your Environment Server setup and configuration: HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw Operating System and version (uname -a): linux vedi 5.10.0-16-amd64 https://github.com/nspcc-dev/neofs-node/issues/1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux