TrueCloudLab/xk6-frostfs
42
1
Fork 20
Code Issues 32 Pull requests 3 Releases Activity Actions

[#136] registry: Apply updates in batch #137

Merged
fyrchik merged 1 commit from fyrchik/xk6-frostfs:optimize-registry into master 2024-05-03 12:25:34 +00:00
Conversation 1 Commits 1 Files changed 1 +1 -1
fyrchik commented 2024-05-03 11:00:33 +00:00
Owner
Copy link

Before

service@loader-167:/etc/k6$ /etc/k6/k6 run -e NO_VERIFY_SSL='True' \
-e DURATION='600' -e WRITE_OBJ_SIZE='8' \
-e REGISTRY_FILE='/var/log/autotests/s3_write_8h_8KB-TEST.bolt' \
-e K6_SETUP_TIMEOUT='5s' -e WRITERS='600' -e READERS='0' -e DELETERS='0' \
-e PREGEN_JSON='/var/log/autotests/s3_02052024_13h31m00s_0_prepare.json' \
-e S3_ENDPOINTS='https://10.78.70.152,https://10.78.70.155,https://10.78.70.153,https://10.78.70.154' \
/etc/k6/scenarios/s3.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

WARN[10:30:48] Couldn't load source map for file:///etc/k6/scenarios/libs/k6-utils-1.4.0.js  error="The moduleSpecifier \"file:///etc/k6/scenarios/libs/index.js.map\" couldn't be found on local disk. Make sure that you've specified the right path to the file. If you're running k6 using the Docker image make sure you have mounted the local directory (-v /local/path/:/inside/docker/path) containing your script and modules so that they're accessible by k6 from inside of the container, see https://k6.io/docs/using-k6/modules#using-local-modules-with-docker."
  execution: local
     script: /etc/k6/scenarios/s3.js
     output: -

  scenarios: (100.00%) 1 scenario, 600 max VUs, 10m5s max duration (incl. graceful stop):
           * write: 600 looping VUs for 10m0s (exec: obj_write, gracefulStop: 5s)

INFO[10:30:48] Pregenerated buckets:          100            source=console
INFO[10:30:48] Pregenerated read object size: 8 Kb           source=console
INFO[10:30:48] Pregenerated total objects:    0              source=console
INFO[10:30:48] Reading VUs:                   0              source=console
INFO[10:30:48] Writing VUs:                   600            source=console
INFO[10:30:48] Deleting VUs:                  0              source=console
INFO[10:30:48] Total VUs:                     600            source=console
INFO[10:30:48] Load started at:               Fri May 03 2024 10:30:48 GMT+0000 (UTC)  source=console
INFO[10:40:48] Load finished at:              Fri May 03 2024 10:40:48 GMT+0000 (UTC)  source=console
     █ setup

     █ teardown

     aws_obj_put_bytes......: 7.9 GB 13 MB/s
     aws_obj_put_duration...: avg=80.59ms  min=25.47ms med=62.45ms  max=2.57s p(90)=106.59ms p(95)=157.62ms
     aws_obj_put_success....: 967665 1611.832262/s
     data_received..........: 0 B    0 B/s
     data_sent..............: 7.9 GB 13 MB/s
     iteration_duration.....: avg=372.09ms min=3.22ms  med=319.03ms max=2.72s p(90)=618.98ms p(95)=717.62ms
     iterations.............: 967665 1611.832262/s
     vus....................: 600    min=600       max=600

running (10m00.4s), 000/600 VUs, 967665 complete and 0 interrupted iterations
write ✓ [======================================] 600 VUs  10m0s

After

service@loader-167:/etc/k6$ /home/service/estratonikov/k6_batch run -e NO_VERIFY_SSL='True' \
-e DURATION='600' -e WRITE_OBJ_SIZE='8' \
-e REGISTRY_FILE='/var/log/autotests/s3_write_8h_8KB-TEST.bolt' \
-e K6_SETUP_TIMEOUT='5s' -e WRITERS='600' -e READERS='0' -e DELETERS='0' \
-e PREGEN_JSON='/var/log/autotests/s3_02052024_13h31m00s_0_prepare.json' \
-e S3_ENDPOINTS='https://10.78.70.152,https://10.78.70.155,https://10.78.70.153,https://10.78.70.154' \
/etc/k6/scenarios/s3.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

WARN[10:42:16] Couldn't load source map for file:///etc/k6/scenarios/libs/k6-utils-1.4.0.js  error="The moduleSpecifier \"file:///etc/k6/scenarios/libs/index.js.map\" couldn't be found on local disk. Make sure that you've specified the right path to the file. If you're running k6 using the Docker image make sure you have mounted the local directory (-v /local/path/:/inside/docker/path) containing your script and modules so that they're accessible by k6 from inside of the container, see https://k6.io/docs/using-k6/modules#using-local-modules-with-docker."
  execution: local
     script: /etc/k6/scenarios/s3.js
     output: -

  scenarios: (100.00%) 1 scenario, 600 max VUs, 10m5s max duration (incl. graceful stop):
           * write: 600 looping VUs for 10m0s (exec: obj_write, gracefulStop: 5s)

INFO[10:42:16] Pregenerated buckets:          100            source=console
INFO[10:42:16] Pregenerated read object size: 8 Kb           source=console
INFO[10:42:16] Pregenerated total objects:    0              source=console
INFO[10:42:16] Reading VUs:                   0              source=console
INFO[10:42:16] Writing VUs:                   600            source=console
INFO[10:42:16] Deleting VUs:                  0              source=console
INFO[10:42:16] Total VUs:                     600            source=console
INFO[10:42:16] Load started at:               Fri May 03 2024 10:42:16 GMT+0000 (UTC)  source=console
INFO[10:52:17] Load finished at:              Fri May 03 2024 10:52:17 GMT+0000 (UTC)  source=console
     █ setup

     █ teardown

     aws_obj_put_bytes......: 17 GB   29 MB/s
     aws_obj_put_duration...: avg=161.67ms min=34.82ms  med=123.38ms max=3.19s p(90)=278.25ms p(95)=372.43ms
     aws_obj_put_success....: 2112322 3518.524981/s
     data_received..........: 0 B     0 B/s
     data_sent..............: 17 GB   29 MB/s
     iteration_duration.....: avg=170.42ms min=230.48µs med=132.5ms  max=3.2s  p(90)=287.28ms p(95)=381.39ms
     iterations.............: 2112322 3518.524981/s
     vus....................: 600     min=600       max=600

running (10m00.3s), 000/600 VUs, 2112322 complete and 0 interrupted iterations
write ✓ [======================================] 600 VUs  10m0s
Before ``` service@loader-167:/etc/k6$ /etc/k6/k6 run -e NO_VERIFY_SSL='True' \ -e DURATION='600' -e WRITE_OBJ_SIZE='8' \ -e REGISTRY_FILE='/var/log/autotests/s3_write_8h_8KB-TEST.bolt' \ -e K6_SETUP_TIMEOUT='5s' -e WRITERS='600' -e READERS='0' -e DELETERS='0' \ -e PREGEN_JSON='/var/log/autotests/s3_02052024_13h31m00s_0_prepare.json' \ -e S3_ENDPOINTS='https://10.78.70.152,https://10.78.70.155,https://10.78.70.153,https://10.78.70.154' \ /etc/k6/scenarios/s3.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io WARN[10:30:48] Couldn't load source map for file:///etc/k6/scenarios/libs/k6-utils-1.4.0.js error="The moduleSpecifier \"file:///etc/k6/scenarios/libs/index.js.map\" couldn't be found on local disk. Make sure that you've specified the right path to the file. If you're running k6 using the Docker image make sure you have mounted the local directory (-v /local/path/:/inside/docker/path) containing your script and modules so that they're accessible by k6 from inside of the container, see https://k6.io/docs/using-k6/modules#using-local-modules-with-docker." execution: local script: /etc/k6/scenarios/s3.js output: - scenarios: (100.00%) 1 scenario, 600 max VUs, 10m5s max duration (incl. graceful stop): * write: 600 looping VUs for 10m0s (exec: obj_write, gracefulStop: 5s) INFO[10:30:48] Pregenerated buckets: 100 source=console INFO[10:30:48] Pregenerated read object size: 8 Kb source=console INFO[10:30:48] Pregenerated total objects: 0 source=console INFO[10:30:48] Reading VUs: 0 source=console INFO[10:30:48] Writing VUs: 600 source=console INFO[10:30:48] Deleting VUs: 0 source=console INFO[10:30:48] Total VUs: 600 source=console INFO[10:30:48] Load started at: Fri May 03 2024 10:30:48 GMT+0000 (UTC) source=console INFO[10:40:48] Load finished at: Fri May 03 2024 10:40:48 GMT+0000 (UTC) source=console █ setup █ teardown aws_obj_put_bytes......: 7.9 GB 13 MB/s aws_obj_put_duration...: avg=80.59ms min=25.47ms med=62.45ms max=2.57s p(90)=106.59ms p(95)=157.62ms aws_obj_put_success....: 967665 1611.832262/s data_received..........: 0 B 0 B/s data_sent..............: 7.9 GB 13 MB/s iteration_duration.....: avg=372.09ms min=3.22ms med=319.03ms max=2.72s p(90)=618.98ms p(95)=717.62ms iterations.............: 967665 1611.832262/s vus....................: 600 min=600 max=600 running (10m00.4s), 000/600 VUs, 967665 complete and 0 interrupted iterations write ✓ [======================================] 600 VUs 10m0s ``` After ``` service@loader-167:/etc/k6$ /home/service/estratonikov/k6_batch run -e NO_VERIFY_SSL='True' \ -e DURATION='600' -e WRITE_OBJ_SIZE='8' \ -e REGISTRY_FILE='/var/log/autotests/s3_write_8h_8KB-TEST.bolt' \ -e K6_SETUP_TIMEOUT='5s' -e WRITERS='600' -e READERS='0' -e DELETERS='0' \ -e PREGEN_JSON='/var/log/autotests/s3_02052024_13h31m00s_0_prepare.json' \ -e S3_ENDPOINTS='https://10.78.70.152,https://10.78.70.155,https://10.78.70.153,https://10.78.70.154' \ /etc/k6/scenarios/s3.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io WARN[10:42:16] Couldn't load source map for file:///etc/k6/scenarios/libs/k6-utils-1.4.0.js error="The moduleSpecifier \"file:///etc/k6/scenarios/libs/index.js.map\" couldn't be found on local disk. Make sure that you've specified the right path to the file. If you're running k6 using the Docker image make sure you have mounted the local directory (-v /local/path/:/inside/docker/path) containing your script and modules so that they're accessible by k6 from inside of the container, see https://k6.io/docs/using-k6/modules#using-local-modules-with-docker." execution: local script: /etc/k6/scenarios/s3.js output: - scenarios: (100.00%) 1 scenario, 600 max VUs, 10m5s max duration (incl. graceful stop): * write: 600 looping VUs for 10m0s (exec: obj_write, gracefulStop: 5s) INFO[10:42:16] Pregenerated buckets: 100 source=console INFO[10:42:16] Pregenerated read object size: 8 Kb source=console INFO[10:42:16] Pregenerated total objects: 0 source=console INFO[10:42:16] Reading VUs: 0 source=console INFO[10:42:16] Writing VUs: 600 source=console INFO[10:42:16] Deleting VUs: 0 source=console INFO[10:42:16] Total VUs: 600 source=console INFO[10:42:16] Load started at: Fri May 03 2024 10:42:16 GMT+0000 (UTC) source=console INFO[10:52:17] Load finished at: Fri May 03 2024 10:52:17 GMT+0000 (UTC) source=console █ setup █ teardown aws_obj_put_bytes......: 17 GB 29 MB/s aws_obj_put_duration...: avg=161.67ms min=34.82ms med=123.38ms max=3.19s p(90)=278.25ms p(95)=372.43ms aws_obj_put_success....: 2112322 3518.524981/s data_received..........: 0 B 0 B/s data_sent..............: 17 GB 29 MB/s iteration_duration.....: avg=170.42ms min=230.48µs med=132.5ms max=3.2s p(90)=287.28ms p(95)=381.39ms iterations.............: 2112322 3518.524981/s vus....................: 600 min=600 max=600 running (10m00.3s), 000/600 VUs, 2112322 complete and 0 interrupted iterations write ✓ [======================================] 600 VUs 10m0s ```
fyrchik added 1 commit 2024-05-03 11:00:35 +00:00
[#136] registry: Apply updates in batch
All checks were successful
DCO action / DCO (pull_request) Successful in 1m7s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 2m27s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 3m0s
Details
Tests and linters / Tests with -race (pull_request) Successful in 3m19s
Details
Tests and linters / Lint (pull_request) Successful in 4m33s
Details
e5af4112f9 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
dstepanov-yadro approved these changes 2024-05-03 11:21:59 +00:00
acid-ant approved these changes 2024-05-03 11:23:35 +00:00
fyrchik commented 2024-05-03 12:25:29 +00:00
Author
Owner
Copy link

We have also tested this on an empty registry -- it was still faster. I was afraid that batch delay will induce artificial slowdown. It is only the case with a very small number of threads, not something we really test.

We have also tested this on an empty registry -- it was still faster. I was afraid that batch delay will induce artificial slowdown. It is only the case with a very small number of threads, not something we really test.
fyrchik merged commit e5af4112f9 into master 2024-05-03 12:25:34 +00:00
fyrchik deleted branch optimize-registry 2024-05-03 12:25:36 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
dstepanov-yadro
acid-ant
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/xk6-frostfs#137
No description provided.
Delete branch "fyrchik/xk6-frostfs:optimize-registry"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?