[#14]: Add local target support #35

Merged
fyrchik merged 1 commit from ale64bit/xk6-frostfs:feature/14-local_targets into master 2023-03-23 11:58:11 +00:00
Member

Signed-off-by: Alejandro Lopez a.lopez@yadro.com

Close #14

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com> Close #14
ale64bit requested review from storage-core-committers 2023-03-16 09:47:32 +00:00
ale64bit requested review from storage-core-developers 2023-03-16 11:58:24 +00:00
fyrchik reviewed 2023-03-16 17:39:04 +00:00
@ -0,0 +1,24 @@
import {uuidv4} from 'https://jslib.k6.io/k6-utils/1.2.0/index.js';
Owner

Can we clone it? See #19 for scenarios/libs.
Also it feels k6-related, can't we import it without https?

Can we clone it? See #19 for `scenarios/libs`. Also it feels `k6`-related, can't we import it without `https`?
Author
Member

Possibly, but this is what all examples were doing so I copy-pasted it blindly. Let's refactor it in a separate PR since it touches a bunch of other files.

Possibly, but this is what all examples were doing so I copy-pasted it blindly. Let's refactor it in a separate PR since it touches a bunch of other files.
fyrchik marked this conversation as resolved
@ -0,0 +36,4 @@
// ngs maps configuration file names to engine instances, since multiple concurrent instances
// are not possible and it's assumed that each storage instance is used independently. This
// way each VU gets its own client even if the underlying engine is the same.
ngs map[string]*engine.StorageEngine
Owner

We do not need to support multiple configuration files here, 1 is enough.
Or do you have any scenario in mind?

We do not need to support multiple configuration files here, 1 is enough. Or do you have any scenario in mind?
Author
Member

done. Turned it into a singleton. (the original reason was that Connect is called multiple times and you can't have more than an instance of the engine running).

done. Turned it into a singleton. (the original reason was that `Connect` is called multiple times and you can't have more than an instance of the engine running).
fyrchik marked this conversation as resolved
@ -0,0 +26,4 @@
const duration = __ENV.DURATION;
const delete_age = __ENV.DELETE_AGE ? parseInt(__ENV.DELETE_AGE) : undefined;
Owner

Current main usecase is PUT/GET, can we delete DELETE-related stuff for now?

Current main usecase is PUT/GET, can we delete DELETE-related stuff for now?
Author
Member

I see. But it already works :) no harm in keeping it?

I see. But it already works :) no harm in keeping it?
fyrchik marked this conversation as resolved
@ -0,0 +112,4 @@
}
export function obj_write() {
if (__ENV.SLEEP_WRITE) {
Owner

We need SLEEP_* variables for network load, not to overload a system.
It is not applicable here (we have no timeout for operations) and is deprecated anyway.
Can we remove this?

We need `SLEEP_*` variables for network load, not to overload a system. It is not applicable here (we have no timeout for operations) and is deprecated anyway. Can we remove this?
Author
Member

done

done
fyrchik marked this conversation as resolved
@ -0,0 +128,4 @@
return;
}
if (obj_registry) {
Owner

@anikeev-yadro , do we need registry for local load? The only case is to have GET on a loaded cluster, but then we need also to support merging registries from different nodes (or do we do local load on 1 node only?)

@anikeev-yadro , do we need registry for local load? The only case is to have GET on a loaded cluster, but then we need also to support merging registries from different nodes (or do we do local load on 1 node only?)
Member

@fyrchik I think we need to keep the registry. We will think about merge in future.

@fyrchik I think we need to keep the registry. We will think about merge in future.
Author
Member

The registry is used also for e.g. delete workload when there are no pregenerated objects. So probably worth keeping it at least for local tests.

The registry is used also for e.g. delete workload when there are no pregenerated objects. So probably worth keeping it at least for local tests.
fyrchik marked this conversation as resolved
ale64bit force-pushed feature/14-local_targets from 30a61f8e2e to cceccd2ccb 2023-03-17 07:25:43 +00:00 Compare
dstepanov-yadro approved these changes 2023-03-17 09:28:46 +00:00
acid-ant approved these changes 2023-03-20 17:09:03 +00:00
ale64bit force-pushed feature/14-local_targets from cceccd2ccb to 5d0134219c 2023-03-21 06:49:13 +00:00 Compare
ale64bit requested review from dstepanov-yadro 2023-03-21 06:49:43 +00:00
ale64bit requested review from acid-ant 2023-03-21 06:49:46 +00:00
dstepanov-yadro approved these changes 2023-03-21 07:07:23 +00:00
ale64bit force-pushed feature/14-local_targets from 5d0134219c to c7a17bea8f 2023-03-21 07:50:19 +00:00 Compare
Author
Member

Object signature was missing. Added in last push.

Object signature was missing. Added in last push.
acid-ant approved these changes 2023-03-21 08:10:32 +00:00
Member

We have the following errors after PUT objects by local target and start frostfs-storage service:

Mar 21 09:06:20 az frostfs-node[520405]: 2023-03-21T09:06:20.597Z        error        replicator/process.go:62        could not replicate object        {"component": "Object Replicator", "node": "0229e023602d10df1fb1018b9ea569d73ba2ad754fbdc5739a3e9467cd4772b0d1", "object": "2MG5YFertEqwsFpVGgQqwYeaKzq9qLAqHJKEftdbEgcs/BbWxnW3k46qYvp8mH5n3SQpvheAM5xBEH5N3wurkPJgK", "error": "(*putsvc.RemoteSender) could not send object: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.115/tcp/8080 /ip4/192.168.201.135/tcp/8080 /ip4/10.78.70.115/tcp/8080 /ip4/10.78.70.135/tcp/8080]: write object via client: status: code = 1024 message = wrong payload size"}
Mar 21 09:06:20 az frostfs-node[520405]: 2023-03-21T09:06:20.599Z        error        replicator/process.go:62        could not replicate object        {"component": "Object Replicator", "node": "035a33242293e0fdc0be32741072fdd022335d6c5cfc207fd934d8a5857510d7ff", "object": "2MG5YFertEqwsFpVGgQqwYeaKzq9qLAqHJKEftdbEgcs/BdEHs2pDfLrZgbCmSf2c9hNaz9tRoyLnJ1CRWtXHqLYe", "error": "(*putsvc.RemoteSender) could not send object: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.116/tcp/8080 /ip4/192.168.201.136/tcp/8080 /ip4/10.78.70.116/tcp/8080 /ip4/10.78.70.136/tcp/8080]: write object via client: status: code = 1024 message = wrong payload size"}
We have the following errors after PUT objects by local target and start frostfs-storage service: ``` Mar 21 09:06:20 az frostfs-node[520405]: 2023-03-21T09:06:20.597Z error replicator/process.go:62 could not replicate object {"component": "Object Replicator", "node": "0229e023602d10df1fb1018b9ea569d73ba2ad754fbdc5739a3e9467cd4772b0d1", "object": "2MG5YFertEqwsFpVGgQqwYeaKzq9qLAqHJKEftdbEgcs/BbWxnW3k46qYvp8mH5n3SQpvheAM5xBEH5N3wurkPJgK", "error": "(*putsvc.RemoteSender) could not send object: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.115/tcp/8080 /ip4/192.168.201.135/tcp/8080 /ip4/10.78.70.115/tcp/8080 /ip4/10.78.70.135/tcp/8080]: write object via client: status: code = 1024 message = wrong payload size"} Mar 21 09:06:20 az frostfs-node[520405]: 2023-03-21T09:06:20.599Z error replicator/process.go:62 could not replicate object {"component": "Object Replicator", "node": "035a33242293e0fdc0be32741072fdd022335d6c5cfc207fd934d8a5857510d7ff", "object": "2MG5YFertEqwsFpVGgQqwYeaKzq9qLAqHJKEftdbEgcs/BdEHs2pDfLrZgbCmSf2c9hNaz9tRoyLnJ1CRWtXHqLYe", "error": "(*putsvc.RemoteSender) could not send object: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.116/tcp/8080 /ip4/192.168.201.136/tcp/8080 /ip4/10.78.70.116/tcp/8080 /ip4/10.78.70.136/tcp/8080]: write object via client: status: code = 1024 message = wrong payload size"} ```
ale64bit force-pushed feature/14-local_targets from c7a17bea8f to 0cb5b4dc98 2023-03-21 12:10:53 +00:00 Compare
ale64bit requested review from dstepanov-yadro 2023-03-22 08:29:51 +00:00
ale64bit requested review from acid-ant 2023-03-22 08:29:54 +00:00
dstepanov-yadro approved these changes 2023-03-23 09:03:43 +00:00
anikeev-yadro approved these changes 2023-03-23 09:09:15 +00:00
fyrchik approved these changes 2023-03-23 11:57:48 +00:00
fyrchik merged commit 7efb6315dc into master 2023-03-23 11:58:11 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/xk6-frostfs#35
No description provided.