feature/add_multipart #80

Merged
alexvanin merged 1 commit from dkirillov/xk6-frostfs:feature/add_multipart into master 2023-07-26 21:08:04 +00:00
Member
No description provided.
dkirillov self-assigned this 2023-07-05 15:05:08 +00:00
dkirillov force-pushed feature/add_multipart from d598c35eaf to d786edd986 2023-07-05 15:05:57 +00:00 Compare
dkirillov force-pushed feature/add_multipart from d786edd986 to 91f8a2e9bd 2023-07-06 08:19:51 +00:00 Compare
dkirillov changed title from WIP: feature/add_multipart to feature/add_multipart 2023-07-06 08:20:13 +00:00
dkirillov requested review from anikeev-yadro 2023-07-06 08:20:52 +00:00
dkirillov requested review from abereziny 2023-07-06 08:20:53 +00:00
dkirillov requested review from storage-core-committers 2023-07-06 08:20:53 +00:00
dkirillov requested review from storage-core-developers 2023-07-06 08:20:53 +00:00
dkirillov requested review from storage-services-committers 2023-07-06 08:20:53 +00:00
dkirillov requested review from storage-services-developers 2023-07-06 08:20:53 +00:00
dkirillov force-pushed feature/add_multipart from 91f8a2e9bd to ecca1386b4 2023-07-11 13:48:32 +00:00 Compare
abereziny approved these changes 2023-07-11 14:22:43 +00:00
alexvanin requested review from mmalygina 2023-07-12 15:07:43 +00:00
fyrchik reviewed 2023-07-17 07:15:47 +00:00
@ -0,0 +34,4 @@
var res PreGenerateInfo
bucketMap := make(map[string]struct{})
count, err := o.selector.Count()
Owner

As I understand, registry DB can be quite big (order of GiB, even more in JSON). What is the motivation for exporting everything in JSON?

As I understand, registry DB can be quite big (order of GiB, even more in JSON). What is the motivation for exporting everything in JSON?
Author
Member

The initial goal was forming pregen json file that includes objects that were created during multipart upload.

But after #25 this probably will be unnecessary

The initial goal was forming pregen json file that includes objects that were created during multipart upload. But after https://git.frostfs.info/TrueCloudLab/xk6-frostfs/issues/25 this probably will be unnecessary
fyrchik marked this conversation as resolved
@ -0,0 +33,4 @@
const obj_to_exporter = registry.getExporter(obj_to_export_selector);
export function obj_registry_export() {
Owner

Why did you decide to write a separate scenario vs small command-line tool?

Why did you decide to write a separate scenario vs small command-line tool?
Author
Member

It was done the similar way as verifying. But actually, there is no significant reason to do it this way.

I'll redo this with command-line tool then

It was done the similar way as verifying. But actually, there is no significant reason to do it this way. I'll redo this with command-line tool then
fyrchik marked this conversation as resolved
@ -134,6 +134,8 @@ Options (in addition to the common options):
* `DELETE_AGE` - age of object in seconds before which it can not be deleted. This parameter can be used to control how many objects we have in the system under load.
* `SLEEP_DELETE` - time interval (in seconds) between deleting VU iterations.
* `OBJ_NAME` - if specified, this name will be used for all write operations instead of random generation.
* `WRITERS_MULTIPART` - number of VUs performing multipart upload operations.
Owner

Is the goal here to have both "simple" and multipart writers in one scenario?

Is the goal here to have both "simple" and multipart writers in one scenario?
Author
Member

Until we have sequential multipart it very similar for simple put. But I'll separate this because I want to support parallel multipart upload in this PR

Until we have sequential multipart it very similar for simple put. But I'll separate this because I want to support parallel multipart upload in this PR
JuliaKovshova approved these changes 2023-07-17 08:58:14 +00:00
dkirillov changed title from feature/add_multipart to WIP: feature/add_multipart 2023-07-17 09:11:21 +00:00
Author
Member

Mark as WIP to support parallel multipart

Mark as WIP to support parallel multipart
dkirillov force-pushed feature/add_multipart from ecca1386b4 to 2a84d844ac 2023-07-18 10:23:03 +00:00 Compare
dkirillov changed title from WIP: feature/add_multipart to feature/add_multipart 2023-07-18 11:09:07 +00:00
dstepanov-yadro reviewed 2023-07-18 11:57:43 +00:00
@ -0,0 +59,4 @@
return fmt.Errorf("get '%s' flag: %w", formatFlag, err)
}
if format != jsonFormat {
return fmt.Errorf("unknown format '%s', only '%s' is supported", format, jsonFormat)

If only one format is supported, then why this parameter? I think this creates unnecessary complexity.

If only one format is supported, then why this parameter? I think this creates unnecessary complexity.
alexvanin approved these changes 2023-07-18 14:59:49 +00:00
mmalygina approved these changes 2023-07-19 07:44:29 +00:00
mmalygina left a comment
Member
No description provided.
dkirillov force-pushed feature/add_multipart from 2a84d844ac to 7c27a5056e 2023-07-19 07:45:19 +00:00 Compare
alexvanin approved these changes 2023-07-19 07:51:41 +00:00
mmalygina approved these changes 2023-07-19 07:55:29 +00:00
mmalygina approved these changes 2023-07-19 07:55:58 +00:00
fyrchik reviewed 2023-07-19 08:09:23 +00:00
@ -0,0 +1,6 @@
package version
var (
// Version is the RBAC sync module version.
Owner

RBAC?

RBAC?
Author
Member

Whoops

Whoops
fyrchik reviewed 2023-07-19 08:10:13 +00:00
@ -0,0 +38,4 @@
const write_multipart_vu_count = parseInt(__ENV.WRITERS_MULTIPART || '0');
if (write_multipart_vu_count < 1) {
throw 'number of parts (env WRITERS_MULTIPART) to upload in parallel should be greater than 0';
Owner

So we explicitly prohibit having "no multipart" scenarios?

So we explicitly prohibit having "no multipart" scenarios?
Author
Member

Yes, it's a separate scenario

Yes, it's a separate scenario
fyrchik approved these changes 2023-07-19 08:10:21 +00:00
dkirillov force-pushed feature/add_multipart from 7c27a5056e to 50e2f55362 2023-07-19 12:58:11 +00:00 Compare
alexvanin approved these changes 2023-07-19 13:00:56 +00:00
alexvanin merged commit 50e2f55362 into master 2023-07-19 13:01:36 +00:00
alexvanin referenced this pull request from a commit 2023-07-19 13:01:37 +00:00
alexvanin referenced this pull request from a commit 2023-07-19 13:01:37 +00:00
alexvanin deleted branch feature/add_multipart 2023-07-19 13:01:39 +00:00
Sign in to join this conversation.
No reviewers
TrueCloudLab/storage-core-developers
TrueCloudLab/storage-services-developers
No milestone
No project
No assignees
7 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/xk6-frostfs#80
No description provided.