feature/add_multipart #80

Merged
alexvanin merged 1 commits from dkirillov/xk6-frostfs:feature/add_multipart into master 2023-07-26 21:08:04 +00:00
Collaborator
There is no content yet.
dkirillov self-assigned this 2023-07-05 15:05:08 +00:00
dkirillov force-pushed feature/add_multipart from d598c35eaf to d786edd986 2023-07-05 15:05:57 +00:00 Compare
dkirillov force-pushed feature/add_multipart from d786edd986 to 91f8a2e9bd 2023-07-06 08:19:51 +00:00 Compare
dkirillov changed title from WIP: feature/add_multipart to feature/add_multipart 2023-07-06 08:20:13 +00:00
dkirillov requested review from anikeev-yadro 2023-07-06 08:20:52 +00:00
dkirillov requested review from abereziny 2023-07-06 08:20:53 +00:00
dkirillov requested review from storage-core-committers 2023-07-06 08:20:53 +00:00
dkirillov requested review from storage-core-developers 2023-07-06 08:20:53 +00:00
dkirillov requested review from storage-services-committers 2023-07-06 08:20:53 +00:00
dkirillov requested review from storage-services-developers 2023-07-06 08:20:53 +00:00
dkirillov force-pushed feature/add_multipart from 91f8a2e9bd to ecca1386b4 2023-07-11 13:48:32 +00:00 Compare
abereziny approved these changes 2023-07-11 14:22:43 +00:00
alexvanin requested review from mmalygina 2023-07-12 15:07:43 +00:00
fyrchik reviewed 2023-07-17 07:15:47 +00:00
@ -0,0 +34,4 @@
var res PreGenerateInfo
bucketMap := make(map[string]struct{})
count, err := o.selector.Count()

As I understand, registry DB can be quite big (order of GiB, even more in JSON). What is the motivation for exporting everything in JSON?

As I understand, registry DB can be quite big (order of GiB, even more in JSON). What is the motivation for exporting everything in JSON?
Poster
Collaborator

The initial goal was forming pregen json file that includes objects that were created during multipart upload.

But after #25 this probably will be unnecessary

The initial goal was forming pregen json file that includes objects that were created during multipart upload. But after https://git.frostfs.info/TrueCloudLab/xk6-frostfs/issues/25 this probably will be unnecessary
fyrchik marked this conversation as resolved
@ -0,0 +33,4 @@
const obj_to_exporter = registry.getExporter(obj_to_export_selector);
export function obj_registry_export() {

Why did you decide to write a separate scenario vs small command-line tool?

Why did you decide to write a separate scenario vs small command-line tool?
Poster
Collaborator

It was done the similar way as verifying. But actually, there is no significant reason to do it this way.

I'll redo this with command-line tool then

It was done the similar way as verifying. But actually, there is no significant reason to do it this way. I'll redo this with command-line tool then
fyrchik marked this conversation as resolved
@ -134,6 +134,8 @@ Options (in addition to the common options):
* `DELETE_AGE` - age of object in seconds before which it can not be deleted. This parameter can be used to control how many objects we have in the system under load.
* `SLEEP_DELETE` - time interval (in seconds) between deleting VU iterations.
* `OBJ_NAME` - if specified, this name will be used for all write operations instead of random generation.
* `WRITERS_MULTIPART` - number of VUs performing multipart upload operations.

Is the goal here to have both "simple" and multipart writers in one scenario?

Is the goal here to have both "simple" and multipart writers in one scenario?
Poster
Collaborator

Until we have sequential multipart it very similar for simple put. But I'll separate this because I want to support parallel multipart upload in this PR

Until we have sequential multipart it very similar for simple put. But I'll separate this because I want to support parallel multipart upload in this PR
JuliaKovshova approved these changes 2023-07-17 08:58:14 +00:00
dkirillov changed title from feature/add_multipart to WIP: feature/add_multipart 2023-07-17 09:11:21 +00:00
Poster
Collaborator

Mark as WIP to support parallel multipart

Mark as WIP to support parallel multipart
dkirillov force-pushed feature/add_multipart from ecca1386b4 to 2a84d844ac 2023-07-18 10:23:03 +00:00 Compare
dkirillov changed title from WIP: feature/add_multipart to feature/add_multipart 2023-07-18 11:09:07 +00:00
dstepanov-yadro reviewed 2023-07-18 11:57:43 +00:00
@ -0,0 +59,4 @@
return fmt.Errorf("get '%s' flag: %w", formatFlag, err)
}
if format != jsonFormat {
return fmt.Errorf("unknown format '%s', only '%s' is supported", format, jsonFormat)

If only one format is supported, then why this parameter? I think this creates unnecessary complexity.

If only one format is supported, then why this parameter? I think this creates unnecessary complexity.
alexvanin approved these changes 2023-07-18 14:59:49 +00:00
mmalygina approved these changes 2023-07-19 07:44:29 +00:00
mmalygina left a comment
Collaborator
There is no content yet.
dkirillov force-pushed feature/add_multipart from 2a84d844ac to 7c27a5056e 2023-07-19 07:45:19 +00:00 Compare
alexvanin approved these changes 2023-07-19 07:51:41 +00:00
mmalygina approved these changes 2023-07-19 07:55:29 +00:00
mmalygina approved these changes 2023-07-19 07:55:58 +00:00
fyrchik reviewed 2023-07-19 08:09:23 +00:00
@ -0,0 +1,6 @@
package version
var (
// Version is the RBAC sync module version.

RBAC?

RBAC?
Poster
Collaborator

Whoops

Whoops
fyrchik reviewed 2023-07-19 08:10:13 +00:00
@ -0,0 +38,4 @@
const write_multipart_vu_count = parseInt(__ENV.WRITERS_MULTIPART || '0');
if (write_multipart_vu_count < 1) {
throw 'number of parts (env WRITERS_MULTIPART) to upload in parallel should be greater than 0';

So we explicitly prohibit having "no multipart" scenarios?

So we explicitly prohibit having "no multipart" scenarios?
Poster
Collaborator

Yes, it's a separate scenario

Yes, it's a separate scenario
fyrchik approved these changes 2023-07-19 08:10:21 +00:00
dkirillov force-pushed feature/add_multipart from 7c27a5056e to 50e2f55362 2023-07-19 12:58:11 +00:00 Compare
alexvanin approved these changes 2023-07-19 13:00:56 +00:00
alexvanin merged commit 50e2f55362 into master 2023-07-19 13:01:36 +00:00
alexvanin deleted branch feature/add_multipart 2023-07-19 13:01:39 +00:00
Sign in to join this conversation.
No reviewers
TrueCloudLab/storage-core-developers
TrueCloudLab/storage-services-developers
No Milestone
No Assignees
7 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/xk6-frostfs#80
There is no content yet.