metabase: Do not store root info for small objects #1007

dstepanov-yadro · 2024-02-26T08:59:04Z

dstepanov-yadro commented

2024-02-26 08:59:04 +00:00

Now metabase doesn't store split info for small objects.

But select --root implementation changed: now root-objects search is more complex.

Relates #977

Now metabase doesn't store split info for small objects. But `select --root` implementation changed: now root-objects search is more complex. Relates #977

fyrchik commented

2024-02-26 09:06:17 +00:00

So we replace --root with --big-only. How can user to list all objects then (the ones that are accounted for in user metrics)?

So we replace `--root` with `--big-only`. How can user to list all objects then (the ones that are accounted for in `user` metrics)?

dstepanov-yadro force-pushed feat/do_not_store_root_info_for_small_objects from 4baf5a006d to b515b1d107

2024-02-26 15:38:15 +00:00

Compare

dstepanov-yadro force-pushed feat/do_not_store_root_info_for_small_objects from b515b1d107 to 3c6055a0c1

2024-02-26 15:38:34 +00:00

Compare

dstepanov-yadro commented

2024-02-26 15:48:25 +00:00

So we replace --root with --big-only. How can user to list all objects then (the ones that are accounted for in user metrics)?

Fixed

> So we replace `--root` with `--big-only`. How can user to list all objects then (the ones that are accounted for in `user` metrics)? Fixed

dstepanov-yadro requested review from storage-core-committers 2024-02-26 15:51:20 +00:00

dstepanov-yadro requested review from storage-core-developers 2024-02-26 15:51:21 +00:00

fyrchik reviewed 2024-02-26 16:02:47 +00:00

pkg/local_object_storage/metabase/select.go Outdated

					
				@ -184,0 +199,4 @@

					_ = primaryBkt.ForEach(func(objectID, rawObject []byte) error {

						obj := objectSDK.New()

						if obj.Unmarshal(rawObject) != nil {

fyrchik commented

2024-02-26 16:02:27 +00:00

Unmarshaling each object can be very costly, is less expensive check currently possible? (ideally checking whether some key is present).
In theory, we could check field presence from the byte-slice directly, though.

Unmarshaling each object can be very costly, is less expensive check currently possible? (ideally checking whether some key is present). In theory, we could check field presence from the byte-slice directly, though.

fyrchik commented

2024-02-26 16:03:26 +00:00

Splitting primary bucket into 2 is also an option.

dstepanov-yadro commented

2024-02-27 05:53:05 +00:00

Unmarshaling each object can be very costly, is less expensive check currently possible?

I haven't found.
As we have the general way to drop all the indexes I think it is ok to have such search implementation.

> Unmarshaling each object can be very costly, is less expensive check currently possible? I haven't found. As we have the general way to drop all the indexes I think it is ok to have such `search` implementation.