transformer: Allow to provide size hint #188

Merged
fyrchik merged 1 commit from fyrchik/frostfs-sdk-go:transformer-size-hint into master 2023-10-30 12:13:06 +00:00
Owner

For big objects with known size we can optimize allocation patterns
by providing size hint. As with any hint, it does not affect transformer
functionality: slices with capacity > MaxSize are never allocated.

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/transformer
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                                                │     out     │
                                                │   sec/op    │
Transformer/small/no_size_hint-8                  65.44µ ± 3%
Transformer/small/no_size_hint,_with_buffer-8     64.24µ ± 5%
Transformer/small/with_size_hint,_with_buffer-8   58.70µ ± 5%
Transformer/big/no_size_hint-8                    367.8m ± 3%
Transformer/big/no_size_hint,_with_buffer-8       562.7m ± 0%
Transformer/big/with_size_hint,_with_buffer-8     385.6m ± 7%
geomean                                           5.197m

                                                │     out      │
                                                │     B/op     │
Transformer/small/no_size_hint-8                  13.40Ki ± 0%
Transformer/small/no_size_hint,_with_buffer-8     13.40Ki ± 0%
Transformer/small/with_size_hint,_with_buffer-8   13.39Ki ± 0%
Transformer/big/no_size_hint-8                    288.0Mi ± 0%
Transformer/big/no_size_hint,_with_buffer-8       1.390Gi ± 0%
Transformer/big/with_size_hint,_with_buffer-8     288.0Mi ± 0%
geomean                                           2.533Mi

                                                │    out     │
                                                │ allocs/op  │
Transformer/small/no_size_hint-8                  92.00 ± 0%
Transformer/small/no_size_hint,_with_buffer-8     92.00 ± 0%
Transformer/small/with_size_hint,_with_buffer-8   92.00 ± 0%
Transformer/big/no_size_hint-8                    546.5 ± 0%
Transformer/big/no_size_hint,_with_buffer-8       607.5 ± 0%
Transformer/big/with_size_hint,_with_buffer-8     545.5 ± 0%
geomean                                           228.1

Signed-off-by: Evgenii Stratonikov e.stratonikov@yadro.com

For big objects with known size we can optimize allocation patterns by providing size hint. As with any hint, it does not affect transformer functionality: slices with capacity > MaxSize are never allocated. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/transformer cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ out │ │ sec/op │ Transformer/small/no_size_hint-8 65.44µ ± 3% Transformer/small/no_size_hint,_with_buffer-8 64.24µ ± 5% Transformer/small/with_size_hint,_with_buffer-8 58.70µ ± 5% Transformer/big/no_size_hint-8 367.8m ± 3% Transformer/big/no_size_hint,_with_buffer-8 562.7m ± 0% Transformer/big/with_size_hint,_with_buffer-8 385.6m ± 7% geomean 5.197m │ out │ │ B/op │ Transformer/small/no_size_hint-8 13.40Ki ± 0% Transformer/small/no_size_hint,_with_buffer-8 13.40Ki ± 0% Transformer/small/with_size_hint,_with_buffer-8 13.39Ki ± 0% Transformer/big/no_size_hint-8 288.0Mi ± 0% Transformer/big/no_size_hint,_with_buffer-8 1.390Gi ± 0% Transformer/big/with_size_hint,_with_buffer-8 288.0Mi ± 0% geomean 2.533Mi │ out │ │ allocs/op │ Transformer/small/no_size_hint-8 92.00 ± 0% Transformer/small/no_size_hint,_with_buffer-8 92.00 ± 0% Transformer/small/with_size_hint,_with_buffer-8 92.00 ± 0% Transformer/big/no_size_hint-8 546.5 ± 0% Transformer/big/no_size_hint,_with_buffer-8 607.5 ± 0% Transformer/big/with_size_hint,_with_buffer-8 545.5 ± 0% geomean 228.1 ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
fyrchik requested review from storage-core-committers 2023-10-26 15:28:26 +00:00
fyrchik requested review from storage-core-developers 2023-10-26 15:28:27 +00:00
fyrchik requested review from storage-services-committers 2023-10-26 15:28:27 +00:00
fyrchik requested review from storage-services-developers 2023-10-26 15:28:28 +00:00
Author
Owner

This could aleviate perfomance problems with client cut on s3 gw cc @alexvanin @dkirillov

This could aleviate perfomance problems with client cut on s3 gw cc @alexvanin @dkirillov
Author
Owner

It also can be used for #155, to allocate buffers of known size.

It also can be used for #155, to allocate buffers of known size.
dstepanov-yadro approved these changes 2023-10-26 15:41:43 +00:00
fyrchik force-pushed transformer-size-hint from 09f0f4f087 to f21d5c99b8 2023-10-27 07:54:29 +00:00 Compare
Author
Owner

Rebased on master, should fix tests.

Rebased on master, should fix tests.
acid-ant approved these changes 2023-10-30 11:28:06 +00:00
fyrchik merged commit 665e5807bc into master 2023-10-30 12:13:06 +00:00
fyrchik deleted branch transformer-size-hint 2023-10-30 12:13:07 +00:00
Sign in to join this conversation.
No reviewers
TrueCloudLab/storage-core-developers
TrueCloudLab/storage-services-committers
TrueCloudLab/storage-services-developers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-sdk-go#188
No description provided.