[#14] Add s3 local loader #48

ale64bit · 2023-04-07T12:40:24Z

ale64bit commented

2023-04-07 12:40:24 +00:00

Signed-off-by: Alejandro Lopez a.lopez@yadro.com

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>

requested reviews from storage-core-committers, storage-core-developers

2023-04-07 12:41:07 +00:00

requested reviews from storage-services-developers, storage-services-committers, anikeev-yadro

2023-04-07 13:45:45 +00:00

dstepanov-yadro reviewed 2023-04-10 06:57:49 +00:00

internal/s3local/client.go Outdated

					
				@ -0,0 +22,4 @@

				type (

					SuccessOrErrorResponse struct {

						Success bool

dstepanov-yadro commented

2023-04-10 06:57:40 +00:00

Why isn't just an error? Why do we need this type?

Why isn't just an ```error```? Why do we need this type?

ale64bit commented

2023-04-10 07:25:14 +00:00

Because all methods that return this type (or similar) are used in JavaScript side, so a compatible type (and number of return values) must be used.

See e.g. #48/files

Because all methods that return this type (or similar) are used in JavaScript side, so a compatible type (and number of return values) must be used. See e.g. https://git.frostfs.info/TrueCloudLab/xk6-frostfs/pulls/48/files#diff-398422f2017b0242d15c6a478df1a8fec108f4af

dstepanov-yadro marked this conversation as resolved

ale64bit commented

2023-04-10 08:01:56 +00:00

A bit of context:

This PR implements a local k6 loader implementation for S3 use-cases, as described in #14. Specifically, it can be used to stop services in a given cluster and fill each node independently and without network usage.

In order to do so, the S3 gateway layer implementation is used directly, which is why a sufficient implementation of the following is needed and included in this PR:

A minimal layer.FrostFS implementation to forward Put/Get calls directly to the local storage engine.
A tree.ServiceClient implementation to forward tree service calls directly to the local pilorama service.
A static bucket resolver to convert bucket names to container IDs and the accompanying tool for computing the mapping from a preset file.

More info is included about these as code comments.

Additionally, the original local scenario implementation is refactored (into rawclient package in this PR) so that it can be reused in this scenario as well.

A bit of context: This PR implements a local k6 loader implementation for S3 use-cases, as described in #14. Specifically, it can be used to stop services in a given cluster and fill each node independently and without network usage. In order to do so, the [S3 gateway layer](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/branch/master/api/layer/layer.go#L187) implementation is used directly, which is why a sufficient implementation of the following is needed and included in this PR: 1. A minimal [layer.FrostFS](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/branch/master/api/layer/frostfs.go#L135) implementation to forward Put/Get calls directly to the local storage engine. 2. A [tree.ServiceClient](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/branch/master/pkg/service/tree/tree.go#L24) implementation to forward tree service calls directly to the local pilorama service. 3. A static bucket resolver to convert bucket names to container IDs and the accompanying tool for computing the mapping from a preset file. More info is included about these as code comments. Additionally, the original local scenario implementation is refactored (into `rawclient` package in this PR) so that it can be reused in this scenario as well.

dstepanov-yadro approved these changes 2023-04-10 09:52:44 +00:00

fyrchik reviewed 2023-04-11 05:08:05 +00:00

examples/s3.js Outdated

					
				@ -1,5 +1,5 @@

				import {uuidv4} from 'https://jslib.k6.io/k6-utils/1.2.0/index.js';

				import {fail} from 'k6'

				import { uuidv4 } from './scenarios/libs/k6-utils-1.4.0.js';

fyrchik commented

2023-04-11 04:58:25 +00:00

Seems unrelated, can we have a separate commit for this?

ale64bit commented

2023-04-11 08:00:38 +00:00

done. Let's do a separate PR.

fyrchik commented

2023-04-12 06:49:34 +00:00

But why not here, seems like a good addition?

go.mod Outdated

					
				@ -5,3 +5,2 @@

				require (

					git.frostfs.info/TrueCloudLab/frostfs-node v0.22.2-0.20230313113918-4e244686cf03

					git.frostfs.info/TrueCloudLab/frostfs-sdk-go v0.0.0-20230307124721-94476f905599

					git.frostfs.info/TrueCloudLab/frostfs-node v0.22.2-0.20230331063259-ed28ce24cd5e

fyrchik commented

2023-04-11 04:59:02 +00:00

And for this too.

ale64bit commented

2023-04-11 08:01:16 +00:00

done

fyrchik marked this conversation as resolved

internal/local/rawclient/option.go Outdated

					
				@ -0,0 +39,4 @@

				// WithPutHandler sets the hook invoked on completion of Put calls.

				// This is useful for updating metrics or debugging.

				func WithPutHandler(h putHandler) Option { return func(c *config) { c.onPut = h } }

fyrchik commented

2023-04-11 05:01:54 +00:00

Why are these functions written in 1 line and WithKey in 4, what's the difference?

Why are these functions written in 1 line and `WithKey` in 4, what's the difference?

ale64bit commented

2023-04-11 08:02:13 +00:00

done

fyrchik marked this conversation as resolved

internal/local/rawclient/option.go Outdated

					
				@ -0,0 +43,4 @@

				// WithGetHandler sets the hook invoked on completion of Get calls.

				// This is useful for updating metrics or debugging.

				func WithGetHandler(h getHandler) Option { return func(c *config) { c.onGet = h } }

fyrchik commented

2023-04-11 05:02:26 +00:00

Can we avoid using parameters of private types for exported methods?

ale64bit commented

2023-04-11 08:02:20 +00:00

done

fyrchik marked this conversation as resolved

internal/s3local/resolver.go Outdated

					
				@ -0,0 +15,4 @@

				func newFixedBucketResolver(bucketMapping map[string]string) (fixedBucketResolver, error) {

					r := fixedBucketResolver{}

					for bucket, cidStr := range bucketMapping {

						id := &cid.ID{}