[#100] s3: Support workflow with versioned buckets #177

nzinkevich · 2024-11-06T10:04:05Z

nzinkevich commented

2024-11-06 10:04:05 +00:00

Add "versioning" parameter handling in CreateBucket
Add a DeleteVersion method for permanent deletion of bucket version. If version parameter is empty, this method deletes all versions and delete-markers of specified object
Add "bucket_versioned" flag for s3_preset.py for setting percent of versioned buckets in case, when we want to create both versioned and non-versioned buckets. Note, that using --versioning flag is the same as --buckets_versioned 100 and both flags shouldn't be use at the same time

Closes #100

Add "versioning" parameter handling in CreateBucket Add a DeleteVersion method for permanent deletion of bucket version. If `version` parameter is empty, this method deletes all versions and delete-markers of specified object Add "bucket_versioned" flag for `s3_preset.py` for setting percent of versioned buckets in case, when we want to create both versioned and non-versioned buckets. Note, that using `--versioning` flag is the same as `--buckets_versioned 100` and both flags shouldn't be use at the same time Closes #100

nzinkevich self-assigned this 2024-11-06 10:04:06 +00:00

nzinkevich added 2 commits 2024-11-06 10:04:06 +00:00

[#100 ] s3: Support creating of versioned bucket c6f7d51393

Add "versioning" parameter handling in CreateBucket method

Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>

[#100 ] s3: Add a flag for permanent versioned object deletion

DCO action / DCO (pull_request) Successful in 1m14s

Details

Tests and linters / Tests (pull_request) Successful in 2m24s

Details

Tests and linters / Tests with -race (pull_request) Successful in 2m44s

Details

Tests and linters / Lint (pull_request) Successful in 3m6s

Details

2c58cbdd43

Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>

requested reviews from storage-services-committers, storage-services-developers, performance-committers

2024-11-06 10:05:58 +00:00

alexvanin reviewed 2024-11-06 11:24:47 +00:00

internal/s3/client.go Outdated

					
				@ -109,2 +109,3 @@

				func (c *Client) Delete(bucket, key string) DeleteResponse {

				func (c *Client) Delete(bucket, key string, allVersions bool) DeleteResponse {

					start := time.Now()

					if allVersions {

alexvanin commented

2024-11-06 11:24:47 +00:00

Discussed with @nzinkevich internally and decided to put this code into separate DeleteVersion command

Discussed with @nzinkevich internally and decided to put this code into separate `DeleteVersion` command

alexvanin marked this conversation as resolved

alexvanin commented

2024-11-06 11:28:46 +00:00

Next step is to modify presets to create combined buckets.

nzinkevich force-pushed feat/versioned_bucket_ops from 2c58cbdd43 to 7a4a4d441d

2024-11-06 12:30:33 +00:00

Compare

nzinkevich added 1 commit 2024-11-06 13:48:07 +00:00

[#100 ] preset_s3: Add a flag for percent of versioned buckets

DCO action / DCO (pull_request) Successful in 1m27s

Details

Tests and linters / Tests (pull_request) Successful in 2m6s

Details

Tests and linters / Tests with -race (pull_request) Successful in 2m58s

Details

Tests and linters / Lint (pull_request) Successful in 3m3s

Details

b8bc12ae6d

Add flag "--buckets_versioned". Default is 0 (no versioned buckets)

Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>

nzinkevich force-pushed feat/versioned_bucket_ops from b8bc12ae6d to 4d28abd692

2024-11-06 13:48:48 +00:00

Compare

nzinkevich force-pushed feat/versioned_bucket_ops from 4d28abd692 to 253d50637f

2024-11-07 06:11:17 +00:00

Compare

alexvanin requested changes 2024-11-12 12:32:03 +00:00

Dismissed

alexvanin left a comment

We need to decide default behavior of s3 present with DELETE workers.

Now it calls Delete and it leaves delete markers in bucket.

We can call DeleteObjectVersion but it produces version listing, which may affect performance.

Steps to reproduce

$ ./scenarios/preset/preset_s3.py --size 100 --buckets 1 --out av.json --endpoint http://localhost:8084 --preload_obj 0 --versioning=True

$ ./k6 run -e DURATION=10 -e WRITE_OBJ_SIZE=10 -e WRITERS=5 -e S3_ENDPOINTS=http://localhost:8084 -e PREGEN_JSON=../av.json -e REGISTRY_FILE=./db scenarios/s3.js

$ ./k6 run -e DURATION=10 -e DELETERS=10 -e DELETE_AGE=1 -e S3_ENDPOINTS=http://localhost:8084 -e PREGEN_JSON=../av.json -e REGISTRY_FILE=./db scenarios/s3.js

We need to decide default behavior of s3 present with DELETE workers. Now it calls `Delete` and it leaves delete markers in bucket. We can call `DeleteObjectVersion` but it produces version listing, which may affect performance. Steps to reproduce ``` $ ./scenarios/preset/preset_s3.py --size 100 --buckets 1 --out av.json --endpoint http://localhost:8084 --preload_obj 0 --versioning=True $ ./k6 run -e DURATION=10 -e WRITE_OBJ_SIZE=10 -e WRITERS=5 -e S3_ENDPOINTS=http://localhost:8084 -e PREGEN_JSON=../av.json -e REGISTRY_FILE=./db scenarios/s3.js $ ./k6 run -e DURATION=10 -e DELETERS=10 -e DELETE_AGE=1 -e S3_ENDPOINTS=http://localhost:8084 -e PREGEN_JSON=../av.json -e REGISTRY_FILE=./db scenarios/s3.js ```

internal/s3/client.go Outdated

					
				@ -145,0 +153,4 @@

							VersionId: aws.String(version),

						})

					} else {

						v, err := c.cli.GetBucketVersioning(c.vu.Context(), &s3.GetBucketVersioningInput{

alexvanin commented

2024-11-12 12:29:13 +00:00

I think we need to avoid calling GetBucketVersioning on every request, because K6 mostly works with the limited number of buckets. It is overhead to call it on every object.

I think we can invoke ListObjectVersions. Then invoke DeleteObjects and pass version id if it is not null. And that's it.

I think we need to avoid calling `GetBucketVersioning` on every request, because K6 mostly works with the limited number of buckets. It is overhead to call it on every object. I think we can invoke `ListObjectVersions`. Then invoke `DeleteObjects` and pass version id if it is not null. And that's it.

alexvanin marked this conversation as resolved

scenarios/preset/preset_s3.py Outdated

					
				@ -28,2 +29,3 @@

				parser.add_argument('--location', help=f'AWS location constraint. Default is "{DEFAULT_LOCATION}"', action="append")

				parser.add_argument('--versioning', help='True/False, False by default.')

				parser.add_argument('--versioning', help='True/False, False by default. Alias of --buckets_versioned=100')

				parser.add_argument('--buckets_versioned', help='Percent of versioned buckets. Default is 0', default=0)

alexvanin commented

2024-11-12 12:21:19 +00:00

Mention new parameters in run_scenarios.md

Mention new parameters in `run_scenarios.md`

alexvanin marked this conversation as resolved

nzinkevich force-pushed feat/versioned_bucket_ops from 253d50637f to 1d37df50fc

2024-11-12 15:21:02 +00:00

Compare

nzinkevich force-pushed feat/versioned_bucket_ops from 1d37df50fc to 124397578d

2024-11-12 15:21:39 +00:00

Compare

alexvanin approved these changes 2024-11-13 15:10:30 +00:00

alexvanin left a comment

With @anikeev-yadro and @a.bogatyrev we've decided to keep Delete invocation in s3 scenarios for both versioned and non-versioned buckets. DeleteObjectVersion may be used to clean-up buckets before bucket removal, but for regular tests we use direct Delete with no extra listings.

With @anikeev-yadro and @a.bogatyrev we've decided to keep `Delete` invocation in s3 scenarios for both versioned and non-versioned buckets. `DeleteObjectVersion` may be used to clean-up buckets before bucket removal, but for regular tests we use direct `Delete` with no extra listings.

alexvanin merged commit 124397578d into master

2024-11-13 15:14:16 +00:00

alexvanin deleted branch feat/versioned_bucket_ops

2024-11-13 15:14:21 +00:00

Sign in to join this conversation.

No reviewers

TrueCloudLab/storage-services-developers

TrueCloudLab/performance-committers