k6 read, delete scenario doesn't stop #149

New issue

Closed

opened 2024-06-20 12:34:13 +00:00 by mmalygina · 3 comments

mmalygina commented

2024-06-20 12:34:13 +00:00

Member

Start k6 scenario with 30s duration

/etc/k6/customk6/xk6-frostfs/k6 run --verbose -e NO_VERIFY_SSL='True' -e DURATION='30' \
-e WRITE_OBJ_SIZE='524288' \
-e REGISTRY_FILE='/var/log/autotests/s3_write_512MB.bolt' \
-e K6_SETUP_TIMEOUT='5s' \
-e WRITERS='0' -e DELETERS='20' -e READERS='20' -e DELETE_AGE="30" \
-e PREGEN_JSON='/var/log/autotests/s3__prepare.json' \
-e S3_ENDPOINTS='https://10.78.70.135,https://10.78.70.136,https://10.78.70.137,https://10.78.70.138' \
/etc/k6/customk6/xk6-frostfs/scenarios/s3_dar.js

Note: I changed line

obj_to_delete_exit_on_null = write_vu_count == 0;

to following - to use registry.getSelector

obj_to_delete_exit_on_null = (write_vu_count == 0) && (read_vu_count == 0);

After 40 seconds it's still running

DEBU[12:30:55] Regular duration is done, waiting for iterations to gracefully finish  executor=constant-vus gracefulStop=5s scenario=read
DEBU[12:30:55] Regular duration is done, waiting for iterations to gracefully finish  executor=constant-vus gracefulStop=5s scenario=delete

running (44.2s), 40/40 VUs, 0 complete and 0 interrupted iterations
delete ? [======================================] 20 VUs  30s
read   ? [======================================] 20 VUs  30s

Start k6 scenario with 30s duration ``` /etc/k6/customk6/xk6-frostfs/k6 run --verbose -e NO_VERIFY_SSL='True' -e DURATION='30' \ -e WRITE_OBJ_SIZE='524288' \ -e REGISTRY_FILE='/var/log/autotests/s3_write_512MB.bolt' \ -e K6_SETUP_TIMEOUT='5s' \ -e WRITERS='0' -e DELETERS='20' -e READERS='20' -e DELETE_AGE="30" \ -e PREGEN_JSON='/var/log/autotests/s3__prepare.json' \ -e S3_ENDPOINTS='https://10.78.70.135,https://10.78.70.136,https://10.78.70.137,https://10.78.70.138' \ /etc/k6/customk6/xk6-frostfs/scenarios/s3_dar.js ``` Note: I changed line https://git.frostfs.info/TrueCloudLab/xk6-frostfs/src/commit/3bc1229062cf44e08171f00ab74a62e40a960d5b/scenarios/s3_dar.js#L78 to following - to use registry.getSelector ``` obj_to_delete_exit_on_null = (write_vu_count == 0) && (read_vu_count == 0); ``` After 40 seconds it's still running ``` DEBU[12:30:55] Regular duration is done, waiting for iterations to gracefully finish executor=constant-vus gracefulStop=5s scenario=read DEBU[12:30:55] Regular duration is done, waiting for iterations to gracefully finish executor=constant-vus gracefulStop=5s scenario=delete running (44.2s), 40/40 VUs, 0 complete and 0 interrupted iterations delete ? [======================================] 20 VUs 30s read ? [======================================] 20 VUs 30s ```

mmalygina commented

2024-06-20 14:37:49 +00:00

Author

Member

Read scenario also doesn't stop

/etc/k6/customk6/xk6-frostfs/k6 run --verbose -e NO_VERIFY_SSL='True' -e DURATION='60' \
-e WRITE_OBJ_SIZE='524288' \
-e REGISTRY_FILE='/var/log/autotests/s3_write_512MB.bolt' \
-e K6_SETUP_TIMEOUT='5s' \
-e WRITERS='0' -e DELETERS='0' -e READERS='20' -e DELETE_AGE="0" \
-e PREGEN_JSON='/var/log/autotests/s3__prepare.json' \
-e S3_ENDPOINTS='https://10.78.70.135,https://10.78.70.136,https://10.78.70.137,https://10.78.70.138' \
/etc/k6/customk6/xk6-frostfs/scenarios/s3_dar.js

Still running after 1 minute passed

INFO[14:33:23] Load started at:               Thu Jun 20 2024 14:33:23 GMT+0000 (UTC)  source=console
DEBU[14:33:23] Start all executors...                        phase=execution-scheduler-run
DEBU[14:33:23] Starting executor                             executor=read startTime=0s type=constant-vus
DEBU[14:33:23] Starting executor run...                      duration=1m0s executor=constant-vus scenario=read type=constant-vus vus=20
DEBU[14:34:23] Regular duration is done, waiting for iterations to gracefully finish  executor=constant-vus gracefulStop=5s scenario=read

running (3m43.9s), 20/20 VUs, 0 complete and 0 interrupted iterations
read ✓ [======================================] 20 VUs  1m0s

Read scenario also doesn't stop ``` /etc/k6/customk6/xk6-frostfs/k6 run --verbose -e NO_VERIFY_SSL='True' -e DURATION='60' \ -e WRITE_OBJ_SIZE='524288' \ -e REGISTRY_FILE='/var/log/autotests/s3_write_512MB.bolt' \ -e K6_SETUP_TIMEOUT='5s' \ -e WRITERS='0' -e DELETERS='0' -e READERS='20' -e DELETE_AGE="0" \ -e PREGEN_JSON='/var/log/autotests/s3__prepare.json' \ -e S3_ENDPOINTS='https://10.78.70.135,https://10.78.70.136,https://10.78.70.137,https://10.78.70.138' \ /etc/k6/customk6/xk6-frostfs/scenarios/s3_dar.js ``` Still running after 1 minute passed ``` INFO[14:33:23] Load started at: Thu Jun 20 2024 14:33:23 GMT+0000 (UTC) source=console DEBU[14:33:23] Start all executors... phase=execution-scheduler-run DEBU[14:33:23] Starting executor executor=read startTime=0s type=constant-vus DEBU[14:33:23] Starting executor run... duration=1m0s executor=constant-vus scenario=read type=constant-vus vus=20 DEBU[14:34:23] Regular duration is done, waiting for iterations to gracefully finish executor=constant-vus gracefulStop=5s scenario=read running (3m43.9s), 20/20 VUs, 0 complete and 0 interrupted iterations read ✓ [======================================] 20 VUs 1m0s ```

mmalygina commented

2024-06-20 15:06:12 +00:00

Author

Member

Problem can be in using SelectorAwaiting for read scenario

fyrchik added the

bug

label 2024-06-21 06:12:45 +00:00

dstepanov-yadro self-assigned this 2024-06-24 06:21:29 +00:00

dstepanov-yadro commented

2024-06-24 06:47:25 +00:00

Member

Hm, everything works on grpc scenario:

#!/bin/bash

xk6 build --with git.frostfs.info/TrueCloudLab/xk6-frostfs=.

echo "Generating preset..."
time /home/dstepanov/src/xk6-frostfs/scenarios/preset/preset_grpc.py \
--size 8 --containers 1 --out /home/dstepanov/src/xk6-frostfs/grpc_preset.json \
--endpoint 127.0.0.1:8080 --preload_obj 0 --wallet /home/dstepanov/src/xk6-frostfs/scenarios/files/wallet.json \
--config empty_pass.yml --policy 'REP 1 CBF 1 SELECT 1 FROM *' --ignore-errors
echo "Preset generated"

echo "Starting grpc write test..."
#K6_PROMETHEUS_RW_SERVER_URL=http://obj-grafana.set.yadro.com:8428/api/v1/write \
#K6_PROMETHEUS_RW_TREND_STATS="p(95),p(99)" \
./k6 run -e DURATION=60 -e WRITERS=5 -e READERS=0 -e DELETERS=0 -e WRITE_OBJ_SIZE=1 \
-e GRPC_ENDPOINTS=127.0.0.1:8080,127.0.0.1:8082,127.0.0.1:8084,127.0.0.1:8086 \
-e PREGEN_JSON=/home/dstepanov/src/xk6-frostfs/grpc_preset.json \
-e REGISTRY_FILE=/home/dstepanov/src/xk6-frostfs/grpc_registry.db \
/home/dstepanov/src/xk6-frostfs/scenarios/grpc.js
echo "grpc write test completed"

echo "Starting grpc read/delete test..."
./k6 run -e DURATION=30 -e WRITERS=0 -e READERS=5 -e DELETERS=5 -e WRITE_OBJ_SIZE=1 -e DELETE_AGE=1 \
-e GRPC_ENDPOINTS=127.0.0.1:8080,127.0.0.1:8082,127.0.0.1:8084,127.0.0.1:8086 \
-e PREGEN_JSON=/home/dstepanov/src/xk6-frostfs/grpc_preset.json \
-e REGISTRY_FILE=/home/dstepanov/src/xk6-frostfs/grpc_registry.db \
/home/dstepanov/src/xk6-frostfs/scenarios/grpc.js
echo "grpc read/delete test completed"

echo "Removing temp files..."
rm grpc_preset.json
rm grpc_registry.db
echo "Temp files removed"

Hm, everything works on grpc scenario: ``` #!/bin/bash xk6 build --with git.frostfs.info/TrueCloudLab/xk6-frostfs=. echo "Generating preset..." time /home/dstepanov/src/xk6-frostfs/scenarios/preset/preset_grpc.py \ --size 8 --containers 1 --out /home/dstepanov/src/xk6-frostfs/grpc_preset.json \ --endpoint 127.0.0.1:8080 --preload_obj 0 --wallet /home/dstepanov/src/xk6-frostfs/scenarios/files/wallet.json \ --config empty_pass.yml --policy 'REP 1 CBF 1 SELECT 1 FROM *' --ignore-errors echo "Preset generated" echo "Starting grpc write test..." #K6_PROMETHEUS_RW_SERVER_URL=http://obj-grafana.set.yadro.com:8428/api/v1/write \ #K6_PROMETHEUS_RW_TREND_STATS="p(95),p(99)" \ ./k6 run -e DURATION=60 -e WRITERS=5 -e READERS=0 -e DELETERS=0 -e WRITE_OBJ_SIZE=1 \ -e GRPC_ENDPOINTS=127.0.0.1:8080,127.0.0.1:8082,127.0.0.1:8084,127.0.0.1:8086 \ -e PREGEN_JSON=/home/dstepanov/src/xk6-frostfs/grpc_preset.json \ -e REGISTRY_FILE=/home/dstepanov/src/xk6-frostfs/grpc_registry.db \ /home/dstepanov/src/xk6-frostfs/scenarios/grpc.js echo "grpc write test completed" echo "Starting grpc read/delete test..." ./k6 run -e DURATION=30 -e WRITERS=0 -e READERS=5 -e DELETERS=5 -e WRITE_OBJ_SIZE=1 -e DELETE_AGE=1 \ -e GRPC_ENDPOINTS=127.0.0.1:8080,127.0.0.1:8082,127.0.0.1:8084,127.0.0.1:8086 \ -e PREGEN_JSON=/home/dstepanov/src/xk6-frostfs/grpc_preset.json \ -e REGISTRY_FILE=/home/dstepanov/src/xk6-frostfs/grpc_registry.db \ /home/dstepanov/src/xk6-frostfs/scenarios/grpc.js echo "grpc read/delete test completed" echo "Removing temp files..." rm grpc_preset.json rm grpc_registry.db echo "Temp files removed" ```

dstepanov-yadro referenced this issue from a pull request that will close it,

2024-06-24 11:01:41 +00:00

[#149] selector: Add read timeout #151

dstepanov-yadro referenced this issue from a commit

2024-06-24 14:45:50 +00:00

[#149] selector: Add read timeout

dstepanov-yadro closed this issue

2024-06-24 14:45:51 +00:00