Preset to generate tree-structured objects in bucket #124

New issue

Closed

opened 2024-01-26 14:14:38 +00:00 by alexvanin · 1 comment

alexvanin commented

2024-01-26 14:14:38 +00:00

Owner

Possible duplicate, close if not relevant.

To showcase FrostFS capabilities, usually storage is loaded with S3 objects in several buckets with scenarios/preset/preset_s3.py. Objects are generated on the root level of bucket, e.g:

- bucket/3a50c948-bda8-4d8d-ab15-b88bd65ec42c
- bucket/b0b343f4-26aa-424f-8b77-3414344802af
- bucket/c542c9b5-6bd3-4cf2-aaa4-32e062e3672e
- bucket/9d6cb5cc-dc69-4c57-976f-bc96ca997590

However this is worst-case scenario for metadata which is stored in a tree. It would be nice to have possibility to generate objects with some dir indent, e.g. Depth and width of the object tree can be parametrized.

- bucket/789b3238-6ced-47a9-8914-3658da32789b/3a50c948-bda8-4d8d-ab15-b88bd65ec42c
- bucket/789b3238-6ced-47a9-8914-3658da32789b/d1ba97fd-c6ae-4bdd-adf8-3fe5bc3b4cd1
- bucket/898fc21a-afee-412c-a766-eb11effe0c46/c542c9b5-6bd3-4cf2-aaa4-32e062e3672e
- bucket/898fc21a-afee-412c-a766-eb11effe0c46/9d6cb5cc-dc69-4c57-976f-bc96ca997590

Possible duplicate, close if not relevant. To showcase FrostFS capabilities, usually storage is loaded with S3 objects in several buckets with `scenarios/preset/preset_s3.py`. Objects are generated on the root level of bucket, e.g: ``` - bucket/3a50c948-bda8-4d8d-ab15-b88bd65ec42c - bucket/b0b343f4-26aa-424f-8b77-3414344802af - bucket/c542c9b5-6bd3-4cf2-aaa4-32e062e3672e - bucket/9d6cb5cc-dc69-4c57-976f-bc96ca997590 ``` However this is worst-case scenario for metadata which is stored in a tree. It would be nice to have possibility to generate objects with some dir indent, e.g. Depth and width of the object tree can be parametrized. ``` - bucket/789b3238-6ced-47a9-8914-3658da32789b/3a50c948-bda8-4d8d-ab15-b88bd65ec42c - bucket/789b3238-6ced-47a9-8914-3658da32789b/d1ba97fd-c6ae-4bdd-adf8-3fe5bc3b4cd1 - bucket/898fc21a-afee-412c-a766-eb11effe0c46/c542c9b5-6bd3-4cf2-aaa4-32e062e3672e - bucket/898fc21a-afee-412c-a766-eb11effe0c46/9d6cb5cc-dc69-4c57-976f-bc96ca997590 ```

fyrchik commented

2024-01-26 14:25:16 +00:00

Owner

We were trying to add these tests and benchmarks to pilorama in the node itself. The problem is that there are too much variables, so you need to know exactly what you want to test.

The problem with using UUIDs on intedimediate levels is that you need to remember them to avoid degenerate cases. Probably sth like bucket/xxx/xxx/3a50c948-bda8-4d8d-ab15-b88bd65ec42c is enough, where xxx is parametrizable (like 3:42 means from 000 to 042). The exact length shouldn't have too much effect. However, the fact that these values are consecutive could have some effect (the keys reside close to each other in DB). So there are tradeoffs.

We were trying to add these tests and benchmarks to pilorama in the node itself. The problem is that there are too much variables, so you need to know _exactly_ what you want to test. The problem with using UUIDs on intedimediate levels is that you need to remember them to avoid degenerate cases. Probably sth like `bucket/xxx/xxx/3a50c948-bda8-4d8d-ab15-b88bd65ec42c` is enough, where `xxx` is parametrizable (like `3:42` means from `000` to `042`). The exact length shouldn't have too much effect. However, the fact that these values are consecutive _could_ have some effect (the keys reside close to each other in DB). So there are tradeoffs.