Verify process stuck if the registry file has 0 objects #66

New Issue

Closed

opened 2023-05-23 12:38:51 +00:00 by anikeev-yadro · 0 comments

anikeev-yadro commented

2023-05-23 12:38:51 +00:00

Collaborator

Expected Behavior

Verify process should end if the registry file has 0 objects.

Current Behavior

Verify process stuck if the registry file has 0 objects.

Steps to Reproduce (for bugs)

1.Do k6 load with 0 putted objects

running (08m04.5s), 00/20 VUs, 909086 complete and 20 interrupted iterations
read  ✗ [  81% ] 10 VUs  08m03.6s/10m0s
write ✗ [  81% ] 10 VUs  08m03.6s/10m0s
     █ setup

     █ teardown

     aws_obj_get_fails....: 904514 1867.070534/s
     aws_obj_put_fails....: 4573   9.439449/s
     data_received........: 0 B    0 B/s
     data_sent............: 0 B    0 B/s
     iteration_duration...: avg=10.61ms min=61.1µs med=2.73ms max=4.38s p(90)=4.07ms p(95)=7.74ms
     iterations...........: 909086 1876.507918/s
     vus..................: 20     min=20        max=20
     vus_max..............: 20     min=20        max=20

2.Start verify

/etc/k6/k6 run -e REGISTRY_FILE='/tmp/s3_e1f6b0ce-1446-4adb-a28b-b44aa151179a_0_registry.bolt' -e TIME_LIMIT='3600' -e CLIENTS='10' -e S3_ENDPOINTS='10.78.70.148:8084,10.78.70.150:8084,10.78.71.148:8084,10.78.70.151:8084,10.78.71.151:8084,10.78.71.150:8084' -e SUMMARY_JSON='/tmp/s3_e1f6b0ce-1446-4adb-a28b-b44aa151179a_0_verify_summary.json' /etc/k6/scenarios/verify.js

3.See that k6 process stuck with 1 VUs



          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: /etc/k6/scenarios/verify.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 1h0m5s max duration (incl. graceful stop):
           * verify: 1 iterations shared among 1 VUs (maxDuration: 1h0m0s, exec: obj_verify, gracefulStop: 5s)


running (0h00m01.0s), 1/1 VUs, 0 complete and 0 interrupted iterations
verify   [   0% ] 1 VUs  0h00m01.0s/1h0m0s  0/1 shared iters

running (0h00m02.0s), 1/1 VUs, 0 complete and 0 interrupted iterations
verify   [   0% ] 1 VUs  0h00m02.0s/1h0m0s  0/1 shared iters

running (0h00m03.0s), 1/1 VUs, 0 complete and 0 interrupted iterations
verify   [   0% ] 1 VUs  0h00m03.0s/1h0m0s  0/1 shared iters
.
.
.
verify ✗ [   0% ] 1 VUs  1h00m05.0s/1h0m0s  0/1 shared iters

running (3h22m30.0s), 1/1 VUs, 0 complete and 0 interrupted iterations
verify ✗ [   0% ] 1 VUs  1h00m05.0s/1h0m0s  0/1 shared iters

running (3h22m31.0s), 1/1 VUs, 0 complete and 0 interrupted iterations
verify ✗ [   0% ] 1 VUs  1h00m05.0s/1h0m0s  0/1 shared iters

running (3h22m32.0s), 1/1 VUs, 0 complete and 0 interrupted iterations
verify ✗ [   0% ] 1 VUs  1h00m05.0s/1h0m0s  0/1 shared iters

4.Kill -9 k6 process and see error log

service@tatlinobject:~$ cat /tmp/proc_4f93c4c1-d46c-4620-aee9-d3f5df521b05/stderr
/tmp/proc_4f93c4c1-d46c-4620-aee9-d3f5df521b05/command.sh: line 7: 3203359 Killed                  /etc/k6/k6 run -e REGISTRY_FILE='/tmp/s3_e1f6b0ce-1446-4adb-a28b-b44aa151179a_0_registry.bolt' -e TIME_LIMIT='3600' -e CLIENTS='10' -e S3_ENDPOINTS='10.78.70.148:8084,10.78.70.150:8084,10.78.71.148:8084,10.78.70.151:8084,10.78.71.151:8084,10.78.71.150:8084' -e SUMMARY_JSON='/tmp/s3_e1f6b0ce-1446-4adb-a28b-b44aa151179a_0_verify_summary.json' /etc/k6/scenarios/verify.js

Your Environment

## Expected Behavior Verify process should end if the registry file has 0 objects. ## Current Behavior Verify process stuck if the registry file has 0 objects. ## Steps to Reproduce (for bugs) 1.Do k6 load with 0 putted objects ``` running (08m04.5s), 00/20 VUs, 909086 complete and 20 interrupted iterations read ✗ [ 81% ] 10 VUs 08m03.6s/10m0s write ✗ [ 81% ] 10 VUs 08m03.6s/10m0s █ setup █ teardown aws_obj_get_fails....: 904514 1867.070534/s aws_obj_put_fails....: 4573 9.439449/s data_received........: 0 B 0 B/s data_sent............: 0 B 0 B/s iteration_duration...: avg=10.61ms min=61.1µs med=2.73ms max=4.38s p(90)=4.07ms p(95)=7.74ms iterations...........: 909086 1876.507918/s vus..................: 20 min=20 max=20 vus_max..............: 20 min=20 max=20 ``` 2.Start verify ``` /etc/k6/k6 run -e REGISTRY_FILE='/tmp/s3_e1f6b0ce-1446-4adb-a28b-b44aa151179a_0_registry.bolt' -e TIME_LIMIT='3600' -e CLIENTS='10' -e S3_ENDPOINTS='10.78.70.148:8084,10.78.70.150:8084,10.78.71.148:8084,10.78.70.151:8084,10.78.71.151:8084,10.78.71.150:8084' -e SUMMARY_JSON='/tmp/s3_e1f6b0ce-1446-4adb-a28b-b44aa151179a_0_verify_summary.json' /etc/k6/scenarios/verify.js ``` 3.See that k6 process stuck with 1 VUs ``` /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io execution: local script: /etc/k6/scenarios/verify.js output: - scenarios: (100.00%) 1 scenario, 1 max VUs, 1h0m5s max duration (incl. graceful stop): * verify: 1 iterations shared among 1 VUs (maxDuration: 1h0m0s, exec: obj_verify, gracefulStop: 5s) running (0h00m01.0s), 1/1 VUs, 0 complete and 0 interrupted iterations verify [ 0% ] 1 VUs 0h00m01.0s/1h0m0s 0/1 shared iters running (0h00m02.0s), 1/1 VUs, 0 complete and 0 interrupted iterations verify [ 0% ] 1 VUs 0h00m02.0s/1h0m0s 0/1 shared iters running (0h00m03.0s), 1/1 VUs, 0 complete and 0 interrupted iterations verify [ 0% ] 1 VUs 0h00m03.0s/1h0m0s 0/1 shared iters . . . verify ✗ [ 0% ] 1 VUs 1h00m05.0s/1h0m0s 0/1 shared iters running (3h22m30.0s), 1/1 VUs, 0 complete and 0 interrupted iterations verify ✗ [ 0% ] 1 VUs 1h00m05.0s/1h0m0s 0/1 shared iters running (3h22m31.0s), 1/1 VUs, 0 complete and 0 interrupted iterations verify ✗ [ 0% ] 1 VUs 1h00m05.0s/1h0m0s 0/1 shared iters running (3h22m32.0s), 1/1 VUs, 0 complete and 0 interrupted iterations verify ✗ [ 0% ] 1 VUs 1h00m05.0s/1h0m0s 0/1 shared iters ``` 4.Kill -9 k6 process and see error log ``` service@tatlinobject:~$ cat /tmp/proc_4f93c4c1-d46c-4620-aee9-d3f5df521b05/stderr /tmp/proc_4f93c4c1-d46c-4620-aee9-d3f5df521b05/command.sh: line 7: 3203359 Killed /etc/k6/k6 run -e REGISTRY_FILE='/tmp/s3_e1f6b0ce-1446-4adb-a28b-b44aa151179a_0_registry.bolt' -e TIME_LIMIT='3600' -e CLIENTS='10' -e S3_ENDPOINTS='10.78.70.148:8084,10.78.70.150:8084,10.78.71.148:8084,10.78.70.151:8084,10.78.71.151:8084,10.78.71.150:8084' -e SUMMARY_JSON='/tmp/s3_e1f6b0ce-1446-4adb-a28b-b44aa151179a_0_verify_summary.json' /etc/k6/scenarios/verify.js ``` ## Your Environment HW