TrueCloudLab/xk6-frostfs
40
0
Fork 19
Code Issues 36 Pull requests 2 Releases Activity Actions

Significant performance regression while writing to registry #167

New issue
Closed
opened 2024-09-06 10:53:25 +00:00 by EliChin · 10 comments
EliChin commented 2024-09-06 10:53:25 +00:00
Member
Copy link

Expected Behavior

Writing to registry shouldn't affect performance results

Current Behavior

While putting objects and writing it to registry, performance drops by half
8KB: 23.5MiB/s 3010.4/s (with registry) VS 33MB/s 4077.2/s (without registry)
1MB: 523.34MiB/s 524.03/s (with registry) VS 1.2 GB/s 1135.7/s (without registry)

Possible Solution

 No fix can be suggested by a QA engineer. Further solutions shall be up to developers.

Steps to Reproduce (for bugs)

  1. create credentials for the user
  2. create buckets: scenarios/preset/preset_s3.py --size 256 --buckets 2 --out preset_bucket.json --endpoint https://data0.funkrock-node1.funkrock.object.set.yadro.com:443 --preload_obj 0 --workers 100 --no-verify-ssl --acl public-read-write
  3. perform load with k6 writing to registry /etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=8 -e WRITERS=600 -e S3_ENDPOINTS=https://data0.funkrock-node1.funkrock.object.set.yadro.com:443,https://data0.funkrock-node2.funkrock.object.set.yadro.com:443,https://data0.funkrock-node3.funkrock.object.set.yadro.com:443,https://data0.funkrock-node4.funkrock.object.set.yadro.com:443,https://data1.funkrock-node1.funkrock.object.set.yadro.com:443,https://data1.funkrock-node2.funkrock.object.set.yadro.com:443,https://data1.funkrock-node3.funkrock.object.set.yadro.com:443,https://data1.funkrock-node4.funkrock.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/preset_bucket.json /etc/k6/scenarios/s3.js -e REGISTRY_FILE=/var/log/autotests/8kb_5min_s3.bolt
  4. perform load with k6 without writing to registry /etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=8 -e WRITERS=600 -e S3_ENDPOINTS=https://data0.funkrock-node1.funkrock.object.set.yadro.com:443,https://data0.funkrock-node2.funkrock.object.set.yadro.com:443,https://data0.funkrock-node3.funkrock.object.set.yadro.com:443,https://data0.funkrock-node4.funkrock.object.set.yadro.com:443,https://data1.funkrock-node1.funkrock.object.set.yadro.com:443,https://data1.funkrock-node2.funkrock.object.set.yadro.com:443,https://data1.funkrock-node3.funkrock.object.set.yadro.com:443,https://data1.funkrock-node4.funkrock.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/preset_bucket.json /etc/k6/scenarios/s3.js
  5. Compare the results

Regression

Yes

Your Environment

  • Version used: latest versions of k6 until 5.09, f539da7
  • Server setup and configuration: FunkRock (4-node Vegman 12+4) v1.6.0-144
## Expected Behavior Writing to registry shouldn't affect performance results ## Current Behavior While putting objects and writing it to registry, performance drops by half 8KB: [23.5MiB/s 3010.4/s (with registry)](https://c.yadro.com/pages/viewpage.action?pageId=1004109897#v1.6.014403092024default(mtlsoff)-Write8KiB30m600thclosedmodel-23.49MiB/s3010.36/s) VS [33MB/s 4077.2/s (without registry)](https://c.yadro.com/pages/viewpage.action?pageId=1004109897#v1.6.014403092024default(mtlsoff)-Write8KiB1h600thclosedmodel-33MB/s4077.2/s) 1MB: [523.34MiB/s 524.03/s (with registry)](https://c.yadro.com/pages/viewpage.action?pageId=1004110204) VS [1.2 GB/s 1135.7/s (without registry)](https://c.yadro.com/pages/viewpage.action?pageId=1004114882) ## Possible Solution No fix can be suggested by a QA engineer. Further solutions shall be up to developers. ## Steps to Reproduce (for bugs) 1. create credentials for the user 2. create buckets: `scenarios/preset/preset_s3.py --size 256 --buckets 2 --out preset_bucket.json --endpoint https://data0.funkrock-node1.funkrock.object.set.yadro.com:443 --preload_obj 0 --workers 100 --no-verify-ssl --acl public-read-write` 3. perform load with k6 writing to registry `/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=8 -e WRITERS=600 -e S3_ENDPOINTS=https://data0.funkrock-node1.funkrock.object.set.yadro.com:443,https://data0.funkrock-node2.funkrock.object.set.yadro.com:443,https://data0.funkrock-node3.funkrock.object.set.yadro.com:443,https://data0.funkrock-node4.funkrock.object.set.yadro.com:443,https://data1.funkrock-node1.funkrock.object.set.yadro.com:443,https://data1.funkrock-node2.funkrock.object.set.yadro.com:443,https://data1.funkrock-node3.funkrock.object.set.yadro.com:443,https://data1.funkrock-node4.funkrock.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/preset_bucket.json /etc/k6/scenarios/s3.js -e REGISTRY_FILE=/var/log/autotests/8kb_5min_s3.bolt` 4. perform load with k6 without writing to registry `/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=8 -e WRITERS=600 -e S3_ENDPOINTS=https://data0.funkrock-node1.funkrock.object.set.yadro.com:443,https://data0.funkrock-node2.funkrock.object.set.yadro.com:443,https://data0.funkrock-node3.funkrock.object.set.yadro.com:443,https://data0.funkrock-node4.funkrock.object.set.yadro.com:443,https://data1.funkrock-node1.funkrock.object.set.yadro.com:443,https://data1.funkrock-node2.funkrock.object.set.yadro.com:443,https://data1.funkrock-node3.funkrock.object.set.yadro.com:443,https://data1.funkrock-node4.funkrock.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/preset_bucket.json /etc/k6/scenarios/s3.js` 5. Compare the results ## Regression Yes ## Your Environment * Version used: latest versions of k6 until 5.09, f539da7 * Server setup and configuration: FunkRock (4-node Vegman 12+4) v1.6.0-144
EliChin added the
bug
 label 2024-09-06 10:53:25 +00:00
fyrchik commented 2024-09-06 11:55:54 +00:00
Owner
Copy link

Refs #141, #136

Refs #141, #136
fyrchik commented 2024-09-06 11:58:18 +00:00
Owner
Copy link

First step is to profile and attach pprof output to this task.

First step is to profile and attach pprof output to this task.
dstepanov-yadro self-assigned this 2024-09-16 14:43:34 +00:00
dstepanov-yadro commented 2024-09-16 15:14:24 +00:00
Member
Copy link

checked on hardware

1 MB, 600th, 5min
without registry:

     data_received..............: 0 B    0 B/s
     data_sent..................: 446 GB 1.5 GB/s
     frostfs_obj_put_bytes......: 446 GB 1.5 GB/s
     frostfs_obj_put_duration...: avg=423.19ms min=43.61ms  med=370.78ms max=2.16s p(90)=835.06ms p(95)=957.75ms
     frostfs_obj_put_success....: 425205 1414.350285/s
     iteration_duration.........: avg=423.65ms min=148.31µs med=371.24ms max=2.16s p(90)=835.55ms p(95)=958.23ms
     iterations.................: 425205 1414.350285/s
     vus........................: 244    min=244       max=600

with registry

     data_received..............: 0 B    0 B/s
     data_sent..................: 454 GB 1.5 GB/s
     frostfs_obj_put_bytes......: 454 GB 1.5 GB/s
     frostfs_obj_put_duration...: avg=402.29ms min=47.05ms  med=356.18ms max=1.9s  p(90)=762.44ms p(95)=873.45ms
     frostfs_obj_put_success....: 433135 1440.678932/s
     iteration_duration.........: avg=415.95ms min=353.24µs med=369.77ms max=1.91s p(90)=776.54ms p(95)=887.7ms 
     iterations.................: 433135 1440.678932/s
     vus........................: 307    min=307       max=600
checked on hardware 1 MB, 600th, 5min without registry: ``` data_received..............: 0 B 0 B/s data_sent..................: 446 GB 1.5 GB/s frostfs_obj_put_bytes......: 446 GB 1.5 GB/s frostfs_obj_put_duration...: avg=423.19ms min=43.61ms med=370.78ms max=2.16s p(90)=835.06ms p(95)=957.75ms frostfs_obj_put_success....: 425205 1414.350285/s iteration_duration.........: avg=423.65ms min=148.31µs med=371.24ms max=2.16s p(90)=835.55ms p(95)=958.23ms iterations.................: 425205 1414.350285/s vus........................: 244 min=244 max=600 ``` with registry ``` data_received..............: 0 B 0 B/s data_sent..................: 454 GB 1.5 GB/s frostfs_obj_put_bytes......: 454 GB 1.5 GB/s frostfs_obj_put_duration...: avg=402.29ms min=47.05ms med=356.18ms max=1.9s p(90)=762.44ms p(95)=873.45ms frostfs_obj_put_success....: 433135 1440.678932/s iteration_duration.........: avg=415.95ms min=353.24µs med=369.77ms max=1.91s p(90)=776.54ms p(95)=887.7ms iterations.................: 433135 1440.678932/s vus........................: 307 min=307 max=600 ```
dstepanov-yadro commented 2024-09-16 15:31:34 +00:00
Member
Copy link

8KB, 600th, 5min

without registry:

     data_received..............: 0 B     0 B/s
     data_sent..................: 10 GB   34 MB/s
     frostfs_obj_put_bytes......: 10 GB   34 MB/s
     frostfs_obj_put_duration...: avg=143.44ms min=11.9ms  med=167.3ms  max=749.73ms p(90)=231.03ms p(95)=245.08ms
     frostfs_obj_put_success....: 1248862 4159.18425/s
     iteration_duration.........: avg=144.14ms min=124.2µs med=167.99ms max=750.41ms p(90)=231.74ms p(95)=245.8ms 
     iterations.................: 1248862 4159.18425/s
     vus........................: 600     min=600      max=600

with registry

     data_received..............: 0 B     0 B/s
     data_sent..................: 10 GB   34 MB/s
     frostfs_obj_put_bytes......: 10 GB   35 MB/s
     frostfs_obj_put_duration...: avg=133.89ms min=12.97ms  med=145.26ms max=666.19ms p(90)=212.89ms p(95)=226.7ms 
     frostfs_obj_put_success....: 1262796 4206.374905/s
     iteration_duration.........: avg=142.54ms min=179.54µs med=154.14ms max=675.24ms p(90)=221.8ms  p(95)=235.23ms
     iterations.................: 1262796 4206.374905/s
     vus........................: 600     min=600       max=600
8KB, 600th, 5min without registry: ``` data_received..............: 0 B 0 B/s data_sent..................: 10 GB 34 MB/s frostfs_obj_put_bytes......: 10 GB 34 MB/s frostfs_obj_put_duration...: avg=143.44ms min=11.9ms med=167.3ms max=749.73ms p(90)=231.03ms p(95)=245.08ms frostfs_obj_put_success....: 1248862 4159.18425/s iteration_duration.........: avg=144.14ms min=124.2µs med=167.99ms max=750.41ms p(90)=231.74ms p(95)=245.8ms iterations.................: 1248862 4159.18425/s vus........................: 600 min=600 max=600 ``` with registry ``` data_received..............: 0 B 0 B/s data_sent..................: 10 GB 34 MB/s frostfs_obj_put_bytes......: 10 GB 35 MB/s frostfs_obj_put_duration...: avg=133.89ms min=12.97ms med=145.26ms max=666.19ms p(90)=212.89ms p(95)=226.7ms frostfs_obj_put_success....: 1262796 4206.374905/s iteration_duration.........: avg=142.54ms min=179.54µs med=154.14ms max=675.24ms p(90)=221.8ms p(95)=235.23ms iterations.................: 1262796 4206.374905/s vus........................: 600 min=600 max=600 ```
dstepanov-yadro commented 2024-09-16 15:34:19 +00:00
Member
Copy link

To estimate registry impact need to check iteration_duration and frostfs_obj_put_bytes (grpc) / aws_obj_put_duration (s3) difference.

@EliChin Please check the results of your tests again: perhaps this is a degradation of the storage, and not a registry.

To estimate registry impact need to check `iteration_duration` and `frostfs_obj_put_bytes` (grpc) / `aws_obj_put_duration` (s3) difference. @EliChin Please check the results of your tests again: perhaps this is a degradation of the storage, and not a registry.
dstepanov-yadro removed their assignment 2024-09-16 15:34:29 +00:00
EliChin was assigned by dstepanov-yadro 2024-09-16 15:34:29 +00:00
EliChin commented 2024-09-18 13:51:07 +00:00
Author
Member
Copy link

@dstepanov-yadro I wonder what commands you used and what your environment was because I've verified my previous results on K6 bcbd0db. In my case writing to registry still leads to significant regression.

Here is the table with the results https://c.yadro.com/display/OBJECT/Table+for+PUT+with+and+without+registry
Please, check K6 pprof profiles attached in the relevant rows of the table

P.S. I will attach commands and full K6 results below

@dstepanov-yadro I wonder what commands you used and what your environment was because I've verified my previous results on K6 bcbd0db. In my case writing to registry still leads to significant regression. Here is the table with the results https://c.yadro.com/display/OBJECT/Table+for+PUT+with+and+without+registry Please, **check K6 pprof profiles** attached in the relevant rows of the **table** P.S. I will attach commands and full K6 results below
EliChin commented 2024-09-18 13:59:28 +00:00
Author
Member
Copy link

8KB 5min 600th s3

without registry

root@loader-211:/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=8 -e WRITERS=600 -e S3_ENDPOINTS=https://data0.funkrock-node1.funkrock.object.set.yadro.com:443,https://data0.funkrock-node2.funkrock.object.set.yadro.com:443,https://data0.funkrock-node3.funkrock.object.set.yadro.com:443,https://data0.funkrock-node4.funkrock.object.set.yadro.com:443,https://data1.funkrock-node1.funkrock.object.set.yadro.com:443,https://data1.funkrock-node2.funkrock.object.set.yadro.com:443,https://data1.funkrock-node3.funkrock.object.set.yadro.com:443,https://data1.funkrock-node4.funkrock.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/preset_100buckets.json /etc/k6/scenarios/s3.js

INFO[13:33:11] Load started at:               Wed Sep 18 2024 13:33:11 GMT+0000 (UTC)  source=console
INFO[13:38:12] Load finished at:              Wed Sep 18 2024 13:38:12 GMT+0000 (UTC)  source=console
     █ setup

     █ teardown

     aws_obj_put_bytes......: 11 GB   37 MB/s
     aws_obj_put_duration...: avg=131.6ms min=28.35ms  med=112.32ms max=5.14s p(90)=210.77ms p(95)=246.67ms
     aws_obj_put_success....: 1363656 4543.394298/s
     data_received..........: 0 B     0 B/s
     data_sent..............: 11 GB   37 MB/s
     iteration_duration.....: avg=132ms   min=141.45µs med=112.73ms max=5.14s p(90)=211.18ms p(95)=247.08ms
     iterations.............: 1363656 4543.394298/s
     vus....................: 600     min=600       max=600

running (5m00.1s), 000/600 VUs, 1363656 complete and 0 interrupted iterations
write ✓ [======================================] 600 VUs  5m0s

with registry

root@loader-211:/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=8 -e WRITERS=600 -e S3_ENDPOINTS=https://data0.funkrock-node1.funkrock.object.set.yadro.com:443,https://data0.funkrock-node2.funkrock.object.set.yadro.com:443,https://data0.funkrock-node3.funkrock.object.set.yadro.com:443,https://data0.funkrock-node4.funkrock.object.set.yadro.com:443,https://data1.funkrock-node1.funkrock.object.set.yadro.com:443,https://data1.funkrock-node2.funkrock.object.set.yadro.com:443,https://data1.funkrock-node3.funkrock.object.set.yadro.com:443,https://data1.funkrock-node4.funkrock.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/preset_100buckets.json /etc/k6/scenarios/s3.js -e REGISTRY_FILE=/var/log/autotests/8kb_5_min_100buckets_s3.bolt

INFO[13:23:41] Load started at:               Wed Sep 18 2024 13:23:41 GMT+0000 (UTC)  source=console
INFO[13:28:42] Load finished at:              Wed Sep 18 2024 13:28:42 GMT+0000 (UTC)  source=console
     █ setup

     █ teardown

     aws_obj_put_bytes......: 1.4 GB 4.5 MB/s
     aws_obj_put_duration...: avg=64.09ms min=29.24ms  med=58.65ms max=1.77s p(90)=65.54ms p(95)=71.4ms
     aws_obj_put_success....: 165487 549.661631/s
     data_received..........: 0 B    0 B/s
     data_sent..............: 1.4 GB 4.5 MB/s
     iteration_duration.....: avg=1.08s   min=384.03µs med=1.09s   max=2.64s p(90)=1.31s   p(95)=1.43s 
     iterations.............: 165487 549.661631/s
     vus....................: 111    min=111      max=600

running (5m01.1s), 000/600 VUs, 165487 complete and 0 interrupted iterations
write ✓ [======================================] 600 VUs  5m0s
8KB 5min 600th s3 ---- without registry ``` root@loader-211:/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=8 -e WRITERS=600 -e S3_ENDPOINTS=https://data0.funkrock-node1.funkrock.object.set.yadro.com:443,https://data0.funkrock-node2.funkrock.object.set.yadro.com:443,https://data0.funkrock-node3.funkrock.object.set.yadro.com:443,https://data0.funkrock-node4.funkrock.object.set.yadro.com:443,https://data1.funkrock-node1.funkrock.object.set.yadro.com:443,https://data1.funkrock-node2.funkrock.object.set.yadro.com:443,https://data1.funkrock-node3.funkrock.object.set.yadro.com:443,https://data1.funkrock-node4.funkrock.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/preset_100buckets.json /etc/k6/scenarios/s3.js ``` ``` INFO[13:33:11] Load started at: Wed Sep 18 2024 13:33:11 GMT+0000 (UTC) source=console INFO[13:38:12] Load finished at: Wed Sep 18 2024 13:38:12 GMT+0000 (UTC) source=console █ setup █ teardown aws_obj_put_bytes......: 11 GB 37 MB/s aws_obj_put_duration...: avg=131.6ms min=28.35ms med=112.32ms max=5.14s p(90)=210.77ms p(95)=246.67ms aws_obj_put_success....: 1363656 4543.394298/s data_received..........: 0 B 0 B/s data_sent..............: 11 GB 37 MB/s iteration_duration.....: avg=132ms min=141.45µs med=112.73ms max=5.14s p(90)=211.18ms p(95)=247.08ms iterations.............: 1363656 4543.394298/s vus....................: 600 min=600 max=600 running (5m00.1s), 000/600 VUs, 1363656 complete and 0 interrupted iterations write ✓ [======================================] 600 VUs 5m0s ``` with registry ``` root@loader-211:/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=8 -e WRITERS=600 -e S3_ENDPOINTS=https://data0.funkrock-node1.funkrock.object.set.yadro.com:443,https://data0.funkrock-node2.funkrock.object.set.yadro.com:443,https://data0.funkrock-node3.funkrock.object.set.yadro.com:443,https://data0.funkrock-node4.funkrock.object.set.yadro.com:443,https://data1.funkrock-node1.funkrock.object.set.yadro.com:443,https://data1.funkrock-node2.funkrock.object.set.yadro.com:443,https://data1.funkrock-node3.funkrock.object.set.yadro.com:443,https://data1.funkrock-node4.funkrock.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/preset_100buckets.json /etc/k6/scenarios/s3.js -e REGISTRY_FILE=/var/log/autotests/8kb_5_min_100buckets_s3.bolt ``` ``` INFO[13:23:41] Load started at: Wed Sep 18 2024 13:23:41 GMT+0000 (UTC) source=console INFO[13:28:42] Load finished at: Wed Sep 18 2024 13:28:42 GMT+0000 (UTC) source=console █ setup █ teardown aws_obj_put_bytes......: 1.4 GB 4.5 MB/s aws_obj_put_duration...: avg=64.09ms min=29.24ms med=58.65ms max=1.77s p(90)=65.54ms p(95)=71.4ms aws_obj_put_success....: 165487 549.661631/s data_received..........: 0 B 0 B/s data_sent..............: 1.4 GB 4.5 MB/s iteration_duration.....: avg=1.08s min=384.03µs med=1.09s max=2.64s p(90)=1.31s p(95)=1.43s iterations.............: 165487 549.661631/s vus....................: 111 min=111 max=600 running (5m01.1s), 000/600 VUs, 165487 complete and 0 interrupted iterations write ✓ [======================================] 600 VUs 5m0s ```
EliChin commented 2024-09-18 14:02:17 +00:00
Author
Member
Copy link

8KB 5min 600th grpc

without registry

root@loader-211:/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITERS=600 -e WRITE_OBJ_SIZE=8 -e GRPC_ENDPOINTS=data0.funkrock-node1.funkrock.object.set.yadro.com:8080,data0.funkrock-node2.funkrock.object.set.yadro.com:8080,data0.funkrock-node3.funkrock.object.set.yadro.com:8080,data0.funkrock-node4.funkrock.object.set.yadro.com:8080,data1.funkrock-node1.funkrock.object.set.yadro.com:8080,data1.funkrock-node2.funkrock.object.set.yadro.com:8080,data1.funkrock-node3.funkrock.object.set.yadro.com:8080,data1.funkrock-node4.funkrock.object.set.yadro.com:8080 -e PREGEN_JSON=/etc/k6/preset_10containers.json /etc/k6/scenarios/grpc.js

INFO[12:46:21] Load started at:               Wed Sep 18 2024 12:46:21 GMT+0000 (UTC)  source=console
INFO[12:51:22] Load finished at:              Wed Sep 18 2024 12:51:22 GMT+0000 (UTC)  source=console
     █ setup

     █ teardown

     data_received..............: 0 B     0 B/s
     data_sent..................: 20 GB   66 MB/s
     frostfs_obj_put_bytes......: 20 GB   66 MB/s
     frostfs_obj_put_duration...: avg=74.09ms min=14.62ms  med=70.1ms  max=420.52ms p(90)=96.26ms p(95)=113.83ms
     frostfs_obj_put_success....: 2403453 8009.122364/s
     iteration_duration.........: avg=74.86ms min=161.63µs med=70.82ms max=421.2ms  p(90)=97.15ms p(95)=114.94ms
     iterations.................: 2403453 8009.122364/s
     vus........................: 600     min=600       max=600
     
running (5m00.1s), 000/600 VUs, 2403453 complete and 0 interrupted iterations
write ✓ [======================================] 600 VUs  5m0s

with registry

root@loader-211:/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITERS=600 -e WRITE_OBJ_SIZE=8 -e GRPC_ENDPOINTS=data0.funkrock-node1.funkrock.object.set.yadro.com:8080,data0.funkrock-node2.funkrock.object.set.yadro.com:8080,data0.funkrock-node3.funkrock.object.set.yadro.com:8080,data0.funkrock-node4.funkrock.object.set.yadro.com:8080,data1.funkrock-node1.funkrock.object.set.yadro.com:8080,data1.funkrock-node2.funkrock.object.set.yadro.com:8080,data1.funkrock-node3.funkrock.object.set.yadro.com:8080,data1.funkrock-node4.funkrock.object.set.yadro.com:8080 -e PREGEN_JSON=/etc/k6/preset_10containers.json /etc/k6/scenarios/grpc.js -e REGISTRY_FILE=/var/log/autotests/8kb_5min_grpc_reg.bolt

INFO[12:53:01] Load started at:               Wed Sep 18 2024 12:53:01 GMT+0000 (UTC)  source=console
INFO[12:58:03] Load finished at:              Wed Sep 18 2024 12:58:03 GMT+0000 (UTC)  source=console
     █ setup

     █ teardown

     data_received..............: 0 B    0 B/s
     data_sent..................: 1.3 GB 4.3 MB/s
     frostfs_obj_put_bytes......: 1.3 GB 4.3 MB/s
     frostfs_obj_put_duration...: avg=41.2ms min=10.27ms med=40.51ms max=169.42ms p(90)=45.25ms p(95)=47.8ms
     frostfs_obj_put_success....: 157872 523.296002/s
     iteration_duration.........: avg=1.14s  min=975.7µs med=1.15s   max=3.02s    p(90)=1.43s   p(95)=1.55s 
     iterations.................: 157872 523.296002/s
     vus........................: 3      min=3        max=600

running (5m01.7s), 000/600 VUs, 157872 complete and 0 interrupted iterations
write ✓ [======================================] 600 VUs  5m0s
8KB 5min 600th grpc --- without registry ``` root@loader-211:/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITERS=600 -e WRITE_OBJ_SIZE=8 -e GRPC_ENDPOINTS=data0.funkrock-node1.funkrock.object.set.yadro.com:8080,data0.funkrock-node2.funkrock.object.set.yadro.com:8080,data0.funkrock-node3.funkrock.object.set.yadro.com:8080,data0.funkrock-node4.funkrock.object.set.yadro.com:8080,data1.funkrock-node1.funkrock.object.set.yadro.com:8080,data1.funkrock-node2.funkrock.object.set.yadro.com:8080,data1.funkrock-node3.funkrock.object.set.yadro.com:8080,data1.funkrock-node4.funkrock.object.set.yadro.com:8080 -e PREGEN_JSON=/etc/k6/preset_10containers.json /etc/k6/scenarios/grpc.js ``` ``` INFO[12:46:21] Load started at: Wed Sep 18 2024 12:46:21 GMT+0000 (UTC) source=console INFO[12:51:22] Load finished at: Wed Sep 18 2024 12:51:22 GMT+0000 (UTC) source=console █ setup █ teardown data_received..............: 0 B 0 B/s data_sent..................: 20 GB 66 MB/s frostfs_obj_put_bytes......: 20 GB 66 MB/s frostfs_obj_put_duration...: avg=74.09ms min=14.62ms med=70.1ms max=420.52ms p(90)=96.26ms p(95)=113.83ms frostfs_obj_put_success....: 2403453 8009.122364/s iteration_duration.........: avg=74.86ms min=161.63µs med=70.82ms max=421.2ms p(90)=97.15ms p(95)=114.94ms iterations.................: 2403453 8009.122364/s vus........................: 600 min=600 max=600 running (5m00.1s), 000/600 VUs, 2403453 complete and 0 interrupted iterations write ✓ [======================================] 600 VUs 5m0s ``` with registry ``` root@loader-211:/etc/k6# /etc/k6/k6 run -e DURATION=300 -e WRITERS=600 -e WRITE_OBJ_SIZE=8 -e GRPC_ENDPOINTS=data0.funkrock-node1.funkrock.object.set.yadro.com:8080,data0.funkrock-node2.funkrock.object.set.yadro.com:8080,data0.funkrock-node3.funkrock.object.set.yadro.com:8080,data0.funkrock-node4.funkrock.object.set.yadro.com:8080,data1.funkrock-node1.funkrock.object.set.yadro.com:8080,data1.funkrock-node2.funkrock.object.set.yadro.com:8080,data1.funkrock-node3.funkrock.object.set.yadro.com:8080,data1.funkrock-node4.funkrock.object.set.yadro.com:8080 -e PREGEN_JSON=/etc/k6/preset_10containers.json /etc/k6/scenarios/grpc.js -e REGISTRY_FILE=/var/log/autotests/8kb_5min_grpc_reg.bolt ``` ``` INFO[12:53:01] Load started at: Wed Sep 18 2024 12:53:01 GMT+0000 (UTC) source=console INFO[12:58:03] Load finished at: Wed Sep 18 2024 12:58:03 GMT+0000 (UTC) source=console █ setup █ teardown data_received..............: 0 B 0 B/s data_sent..................: 1.3 GB 4.3 MB/s frostfs_obj_put_bytes......: 1.3 GB 4.3 MB/s frostfs_obj_put_duration...: avg=41.2ms min=10.27ms med=40.51ms max=169.42ms p(90)=45.25ms p(95)=47.8ms frostfs_obj_put_success....: 157872 523.296002/s iteration_duration.........: avg=1.14s min=975.7µs med=1.15s max=3.02s p(90)=1.43s p(95)=1.55s iterations.................: 157872 523.296002/s vus........................: 3 min=3 max=600 running (5m01.7s), 000/600 VUs, 157872 complete and 0 interrupted iterations write ✓ [======================================] 600 VUs 5m0s ```
EliChin commented 2024-09-20 12:09:12 +00:00
Author
Member
Copy link

On BlackMetal (Compal, 16-nodes) v1.6.0-169 the bug hasn't been reproduced.

root@loader-101:/etc/k6# /etc/k6/k6 run -e DURATION=30 -e WRITE_OBJ_SIZE=8 -e WRITERS=2500 -e S3_ENDPOINTS=https://data0.blackmetal-node1.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node2.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node3.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node4.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node1.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node2.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node3.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node4.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node5.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node6.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node7.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node8.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node5.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node6.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node7.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node8.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node9.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node10.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node11.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node12.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node13.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node14.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node15.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node16.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node9.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node10.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node11.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node12.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node13.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node14.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node15.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node16.blackmetal.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/100buckets.json /etc/k6/scenarios/s3.js -e REGISTRY_FILE=/var/log/autotests/8kb_s3.bolt --out profile

INFO[12:02:12] Load started at:               Fri Sep 20 2024 12:02:12 GMT+0000 (UTC)  source=console
INFO[12:02:42] Load finished at:              Fri Sep 20 2024 12:02:42 GMT+0000 (UTC)  source=console
     █ setup

     █ teardown

     aws_obj_put_bytes......: 2.5 GB 82 MB/s
     aws_obj_put_duration...: avg=236.73ms min=35.99ms  med=182.57ms max=2.48s p(90)=433.78ms p(95)=536.34ms
     aws_obj_put_success....: 304380 9979.209966/s
     data_received..........: 0 B    0 B/s
     data_sent..............: 2.5 GB 82 MB/s
     iteration_duration.....: avg=246.84ms min=233.95µs med=193.29ms max=2.49s p(90)=442.28ms p(95)=544.74ms
     iterations.............: 304380 9979.209966/s
     vus....................: 2500   min=2500      max=2500

running (30.5s), 0000/2500 VUs, 304380 complete and 0 interrupted iterations
write ✓ [======================================] 2500 VUs  30s
On BlackMetal (Compal, 16-nodes) v1.6.0-169 the bug hasn't been reproduced. ``` root@loader-101:/etc/k6# /etc/k6/k6 run -e DURATION=30 -e WRITE_OBJ_SIZE=8 -e WRITERS=2500 -e S3_ENDPOINTS=https://data0.blackmetal-node1.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node2.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node3.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node4.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node1.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node2.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node3.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node4.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node5.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node6.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node7.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node8.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node5.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node6.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node7.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node8.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node9.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node10.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node11.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node12.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node13.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node14.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node15.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node16.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node9.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node10.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node11.blackmetal.object.set.yadro.com:443,https://data0.blackmetal-node12.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node13.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node14.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node15.blackmetal.object.set.yadro.com:443,https://data1.blackmetal-node16.blackmetal.object.set.yadro.com:443 -e PREGEN_JSON=/etc/k6/100buckets.json /etc/k6/scenarios/s3.js -e REGISTRY_FILE=/var/log/autotests/8kb_s3.bolt --out profile ``` ``` INFO[12:02:12] Load started at: Fri Sep 20 2024 12:02:12 GMT+0000 (UTC) source=console INFO[12:02:42] Load finished at: Fri Sep 20 2024 12:02:42 GMT+0000 (UTC) source=console █ setup █ teardown aws_obj_put_bytes......: 2.5 GB 82 MB/s aws_obj_put_duration...: avg=236.73ms min=35.99ms med=182.57ms max=2.48s p(90)=433.78ms p(95)=536.34ms aws_obj_put_success....: 304380 9979.209966/s data_received..........: 0 B 0 B/s data_sent..............: 2.5 GB 82 MB/s iteration_duration.....: avg=246.84ms min=233.95µs med=193.29ms max=2.49s p(90)=442.28ms p(95)=544.74ms iterations.............: 304380 9979.209966/s vus....................: 2500 min=2500 max=2500 running (30.5s), 0000/2500 VUs, 304380 complete and 0 interrupted iterations write ✓ [======================================] 2500 VUs 30s ```
mmalygina commented 2024-10-14 11:38:39 +00:00
Member
Copy link

Writing to registry affects performance results, if registry is on HDD.

Writing to registry affects performance results, if registry is on HDD.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
lorem-ipsum
No results found.
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
EliChin
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/xk6-frostfs#167
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?