Use streams to marshal/unmarshal object's payload #849

New issue

Open

opened 2023-12-07 06:46:41 +00:00 by acid-ant · 2 comments

acid-ant commented

2023-12-07 06:46:41 +00:00

Member

Problem

In scope of 1338 introduced new argument '--binary' which help us to serialize object structure.
But we need a way to avoid store whole payload in memory to use Marshal/Unmarhsal methods.

Solution

It is preferable to use streams of bytes.

Let's also think about being forwards and/or backwards compatible with the next API versions.

Places, where we need to improve:

objToStore.SetPayload(payloadBuffer.Bytes())

commonCmd.ExitOnErr(cmd, "can't unmarshal object from given file: %w", objTemp.Unmarshal(buf))

**Problem** In scope of [1338](https://github.com/nspcc-dev/neofs-node/issues/1338) introduced new argument '--binary' which help us to serialize object structure. But we need a way to avoid store whole payload in memory to use Marshal/Unmarhsal methods. **Solution** It is preferable to use streams of bytes. Let's also think about being forwards and/or backwards compatible with the next API versions. Related to https://github.com/nspcc-dev/neofs-node/issues/1932 Places, where we need to improve: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/a478050639aac5bb144f5d41b366e26c8956e3c0/cmd/frostfs-cli/modules/object/get.go#L112 https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/a478050639aac5bb144f5d41b366e26c8956e3c0/cmd/frostfs-cli/modules/object/put.go#L160

acid-ant added the

triage

label 2023-12-07 06:46:41 +00:00

acid-ant referenced this issue

2023-12-07 06:47:21 +00:00

Drop/resolve TODO's #847

acid-ant referenced this issue

2023-12-07 06:47:30 +00:00

Drop/resolve TODO's #847

~~acid-ant referenced this issue 2023-12-07 06:48:36 +00:00~~

Drop/resolve TODO's #847

~~acid-ant referenced this issue 2023-12-07 06:49:03 +00:00~~

Drop/resolve TODO's #847

fyrchik commented

2023-12-07 07:14:07 +00:00

Owner

TBH I would rather drop the whole binary feature, it seems unused and storing big objects as --binary actually creates new object, which couldn't be put without further processing (this was the whole motivation behind --binary).

TBH I would rather drop the whole binary feature, it seems unused and storing big objects as `--binary` actually creates new object, which _couldn't be put_ without further processing (this was the whole motivation behind `--binary`).

acid-ant commented

2023-12-07 07:44:30 +00:00

Author

Member