Do not store compressed uncompressible data #209

New issue

Closed

opened 2023-04-04 11:05:24 +00:00 by fyrchik · 3 comments

fyrchik commented

2023-04-04 11:05:24 +00:00

Owner

Zstd does not allow calculating supposed compressed data length.
However, if compressed data is bigger than uncompressed one, we can store uncompressed, should save us storage space and CPU cycles on decoding.

Zstd does not allow calculating supposed compressed data length. However, if compressed data is bigger than uncompressed one, we can store uncompressed, should save us storage space and CPU cycles on decoding.

fyrchik added the

frostfs-node

triage

labels 2023-04-04 11:05:24 +00:00

realloc commented

2023-04-04 11:09:23 +00:00

Owner

Should we "try" compressing some amount of payload and stop doing it when we see it makes no sense?

fyrchik commented

2023-04-04 11:15:50 +00:00

Author

Owner

I was thinking about using sync.Pool to avoid allocations, but first benchmarks showed no improvements.

I am not sure it will be faster, our objects have maximum size ~100MiB, synchronization efforts and trying to mess with zstd internals (which is parallel) can easily make it slower and harder to maintain. I was thinking about using `sync.Pool` to avoid allocations, but first benchmarks showed no improvements.

fyrchik added this to the v0.37.0 milestone 2023-04-11 08:20:55 +00:00

fyrchik removed the

triage

label 2023-04-14 09:18:05 +00:00

fyrchik referenced this issue from a commit

2023-05-04 06:35:50 +00:00

[#209] compression: Do not store uncompressible data

fyrchik commented

2023-05-04 07:23:36 +00:00

Author

Owner

Closed via #297