TrueCloudLab/frostfs-node

Fork 27

One of the nodes at the start of the load has a high frostfs_node_engine_put_duration #281

New issue

Closed

opened 2023-04-25 09:29:39 +00:00 by a.bogatyrev · 4 comments

a.bogatyrev commented

2023-04-25 09:29:39 +00:00

Owner

As you can see on the last run, this state alternates.

Load profile: 8kb for 40 buckets, 800 threads.

At the start of the load (8kb) in a series of launches, unexpected behavior is observed in the frostfs_node_engine_put_duration metric. Each time one of the nodes has a value 3-10 times higher than the other nodes. As you can see on the last run, this state alternates. Load profile: 8kb for 40 buckets, 800 threads.

Screenshot from 2023-04-25 11-55-27.png

67 KiB

a.bogatyrev added the

triage

label 2023-04-25 09:29:39 +00:00

fyrchik commented

2023-04-25 09:45:36 +00:00

Owner

Write-cache takes some time to initialize, could be it.

fyrchik commented

2023-04-25 09:46:36 +00:00

Owner

Oh, stop it is the start of the load, not service, right?

a.bogatyrev commented

2023-04-25 09:57:48 +00:00

Author

Owner

Yes, the run starts on clean nodes (without meta and data)

snegurochka added the

bug

label 2023-05-03 17:14:41 +00:00

fyrchik added the

observability

label 2023-05-18 10:45:26 +00:00

fyrchik added

frostfs-node

and removed

triage

labels 2023-12-06 07:57:08 +00:00

fyrchik added this to the v0.38.0 milestone 2023-12-06 07:57:13 +00:00

dstepanov-yadro self-assigned this 2023-12-21 09:00:54 +00:00

dstepanov-yadro commented

2023-12-22 14:52:21 +00:00

Member

Metric frostfs_node_engine_put_duration was replaced with frostfs_node_engine_request_duration_seconds
Tested on hardware, not reproduced:

There are some pikes, but it is regular values, not 3-10 time bigger.

Metric `frostfs_node_engine_put_duration` was replaced with `frostfs_node_engine_request_duration_seconds` Tested on hardware, not reproduced: ![image](/attachments/2314c88e-672c-4235-9272-29ac9f8de52c) ![image](/attachments/ddb035e0-d22d-48b6-b518-1b6144254870) ![image](/attachments/80aae671-f2af-4e78-a263-641de654522b) ![image](/attachments/2935c236-f318-4e26-9e17-b08bc923c79c) There are some pikes, but it is regular values, not 3-10 time bigger.

image.png

53 KiB

image.png

27 KiB

image.png

42 KiB

image.png

32 KiB

dstepanov-yadro closed this issue

2023-12-22 14:52:21 +00:00