Commit graph

41 commits

Author SHA1 Message Date
b66b5a2f37 [#42] registry: Optimize Count() for ObjSelector
It is the heaviest function executing on setup stage.
The culprit is the linear dependency between its execution
time and the amount of objects in registry. The solution is to store
object by status. While the optimization doesn't work for objects with
no status, it is currently provided by all scenarios.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-15 16:08:26 +03:00
22d7996f79 [#42] registry: Optimize ObjectInfo marshaling
1. Get rid of JSON in the database.
2. Store `CreatedAt` as int64. It decreases JSON marshaling time by
   about ~25% with no changes for native scheme.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-15 16:08:26 +03:00
81b7d3f536 Check NOFILE limit before creating local storage engine
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-21 14:01:30 +03:00
5ab7cfed7c [#52] Add debug_logger flag to local scenarios
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-19 10:44:45 +03:00
de6747fc0f [#14] Add s3 local loader
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-13 13:00:38 +00:00
cea265a3f8 [#43] generator: Fix initial payload generation
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-24 07:29:32 +00:00
7efb6315dc [#14]: Add local target support
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-23 11:58:11 +00:00
ed58333aba [#11] Support reading env variables from file
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-15 09:13:34 +03:00
b1f7b29848 [TrueCloudLab#22] datagen: Use local randomness source
`math/rand.Read` is deprecated.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 06:03:38 +00:00
e96d83549b [TrueCloudLab#22] datagen: Allocate buffer once
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 06:03:38 +00:00
e82b1ebd1d Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 17:53:17 +03:00
4ea872d6c3 [#9] logging: Make logger more functional
Previous iteration was bad, because `Logger` instance was shared and
endpoints in different VUs were overriding each other.

New interface is much better, it supports logger extension in any
context with arbitrary fields.
```
const l = logging.new().withFields({endpoint: "my endpoint"});
...
l.withField("cid", container).info("not found");
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-28 14:18:53 +03:00
4a2b01f5a4 [#8] internal: Add full timestamp and endpoint to logs
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-03 17:12:26 +03:00
e52a315752 [#50] Use native.connect timeouts everywhere
Also, use defaults from the SDK itself, allow user to provide 0.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-03 17:12:26 +03:00
b24baf06ca fix format %w
Signed-off-by: Andrey Berezin <a.berezin@yadro.com>
2023-01-11 20:10:57 +03:00
43fddcb28b sys.exit with exit codes if no containers/objects during preset
correct statistics - total counts of request contains only successful requests.
add endpoint to grpc connection error message
added ability to change workers count during preset

Signed-off-by: Andrey Berezin <a.berezin@yadro.com>
2022-12-30 17:30:50 +03:00
171327cdda [#2] Deep rebranding
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-30 11:05:20 +03:00
88b2129b54 [#50] Allow to set timeouts for native scenario
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
(cherry picked from commit 1337eed6df45492f1045541021e8b14a42c5a286)
2022-12-29 17:35:13 +03:00
Denis Kirillov
23feb6c937 [#44] Add params to disable tls check on s3
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
(cherry picked from commit cb21178ae7e1ba6e956b3b3ea0a845fe6b0ca48a)
2022-12-29 17:35:13 +03:00
966fee0e55 [#1] Build K6 extension with FrostFS dependencies
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-22 11:13:47 +03:00
Denis Kirillov
f6695f3b4c [#29] Add delete metrics
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-12-02 13:05:04 +03:00
883c3c259a [#32] registry: Allow to customize cache size for DELETE
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-11-11 10:26:27 +03:00
Pavel Karpy
c43f73704e [#30] grpc: Cache object getters
Includes:
1. Logic simplification: no need to call `ObjSelector.Reset` from JS code,
everything could be done inside the Go code. Remove unused mutexes.
2. Do not handle object twice ever: Once handled, any error is expected to be
logged on the JS side and never be handled again. It solves "already removed"
error.
3. Object caching: no need to call bolt's `View` on every object removal: it
blocks other calls and slows down the execution. Read 100 objects (or less if
not available yet), cache them and send to buffered channel.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-03 13:11:50 +03:00
Pavel Karpy
1434d95e81 [#30] go.mod: Update neofs-sdk
It fixes panic using go 1.19+.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-03 13:11:50 +03:00
37e27f6791 [#23] Implement deletion of objects
1. Added simple lock mechanism to reset obj selector. This prevents
   most of concurrency issues when multiple VUs try to reset selector.
2. Added logic to delete objects to grpc and s3 scenarios.
3. Added registry support to http scenario.
4. Deletion logic was not implemented for http scenario, because
   http gateway does not provide web-method to delete objects.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-10-03 17:34:20 +03:00
89faf927fb [#21] Improve iteration logic in obj selector
1. Implement reset method that allows to start iteration from beginning of
   the registry. This allows to revisit objects in scenarios like object
   deletion.
2. Add filter structure that allows to select objects based on age.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-30 15:18:45 +03:00
b1ec6d562c [#19] Stop object iteration after all objects were processed
At the moment we don't need logic that swings back to beginning of registry when
all objects have been processed. So, for now we can stop iterating and return an
error when selector reaches the end of registry.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-23 13:36:27 +03:00
bf4a11474e [#19] Add counters to verify scenario
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-23 13:36:27 +03:00
2d4e619992 [#19] Implement configurable database name for registry
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-23 13:36:27 +03:00
1cf53545f2 [#19] Implement objects registry module
Registry module stores information about uploaded objects in bolt database and
allows to verify their validity after a load test.
Also, implemented logic to verify objects uploaded via gRPC and S3 protocols.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-23 13:36:27 +03:00
6b50c8dcf2 [#17] Add unit tests for the generator
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-19 12:57:57 +03:00
962da644af [#17] Implement k6 extension for random data generation
It improves payload generation in our scenarios. Current implementation
of scenarios generates single random payload at the start and then sends this
same payload on every request. More realistic test is to generate unique payload
for each request. However, this is an expensive operation that can easily cause
a bottleneck on K6 side when we run multiple writing VUs. So instead we generate
a random buffer with some extra bytes and then take slices of this buffer thus
producing a random payload for each request.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-19 12:57:57 +03:00
Denis Kirillov
5d77a526d0 [#11] Add container creation
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-18 22:00:52 +03:00
Denis Kirillov
5671b5c73f [#11] Add bucket creation
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-06-18 22:00:52 +03:00
Alex Vanin
faba352e36 [#3] native: Configure buffer size
Default value is increased up to 64 KiB.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-25 18:29:44 +03:00
Alex Vanin
0d1a5e4187 [#6] native: Factor put object upload code
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-23 15:02:47 +03:00
Alex Vanin
2da51e4aa2 [#6] native: Support onsite object preparation and uploading
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-23 15:02:47 +03:00
Alex Vanin
1c0abb2479 [#5] Avoid payload copy in put methods
With goja.ArrayBuffer type VU won't perform
copy operation during every put invocation.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-20 14:33:56 +03:00
Alex Vanin
0cafcd2375 [#4] s3: Include error message in responses
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-20 14:10:53 +03:00
Alex Vanin
822c674c92 [#4] native: Include error message in responses
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-20 14:10:53 +03:00
Alex Vanin
68999e49d4 Initial commit
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-12 23:27:06 +03:00