Commit graph

83 commits

Author SHA1 Message Date
25fbae1582 [#247] object: Introduce patcher package
* Introduce `patcher` package that contains such interfaces to be
  implemented:
  - `PatchApplier` - the main patching engine that merges the stream
    of patches and the stream of original object payload divided by
    ranges. The merged streams result is output to `ChunkedObjectWriter`;
  - `RangeProvider` - provides the original object payload by ranges;
  - `HeaderProvider` - provides the original object header.
* Introduce `patcher` that implements `PatchApplier`;
* Cover all possible cases with unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-02 11:38:22 +03:00
ff563851fc [#247] object: Introduce Patch type
* Make ToV2, FromV2 converters for Patch and PayloadPatch;
* Add unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-02 11:38:18 +03:00
7e94a6adf2 [#237] pool: Return creation epoch from object put
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-07-22 06:15:23 +00:00
ebd8fcd168 [#224] object: Introduce parent attributes in EC-header
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 16:20:48 +03:00
3de256d05e [#223] object: Introduce new fields for ECHeader
* Introduce `parentSplitID`, `parentSplitParentID` fields
  for `ECHeader`;
* Fix ECHeader's constructor;
* Fix `Split` and `Reconstruct`;
* Add unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-17 15:16:28 +03:00
09b79d13f3 [#223] object: Introduce ec_parent search filter
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-17 15:03:40 +03:00
20ab57bf7e [#214] object: Implement Get\Head requests for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-24 11:07:26 +03:00
ec0cb2169f [#211] object: Fix setIDWithSignature
* Calculate and set checksum before ID is calculated.
* Add header verification for parts in unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-29 13:48:04 +03:00
1af9b6d18b [#155] sdk-go: Add buffer support for payloadSizeLimiter
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-03-25 10:47:04 +03:00
bd2d350b09 [#205] object: Initial EC implementation
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-22 10:14:12 +00:00
a86170f53a [#202] object: Reset marshal data on CutPayload
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-15 15:23:47 +03:00
56debcfa56 [#190] sdk-go: Gofumpt fixes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 19:21:20 +03:00
157a9930e8 [#190] sdk-go: Pass user.ID by value
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 19:21:20 +03:00
665e5807bc [#188] transformer: Allow to provide size hint
For big objects with known size we can optimize allocation patterns
by providing size hint. As with any hint, it does not affect transformer
functionality: slices with capacity > MaxSize are never allocated.

```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/transformer
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                                                │     out     │
                                                │   sec/op    │
Transformer/small/no_size_hint-8                  65.44µ ± 3%
Transformer/small/no_size_hint,_with_buffer-8     64.24µ ± 5%
Transformer/small/with_size_hint,_with_buffer-8   58.70µ ± 5%
Transformer/big/no_size_hint-8                    367.8m ± 3%
Transformer/big/no_size_hint,_with_buffer-8       562.7m ± 0%
Transformer/big/with_size_hint,_with_buffer-8     385.6m ± 7%
geomean                                           5.197m

                                                │     out      │
                                                │     B/op     │
Transformer/small/no_size_hint-8                  13.40Ki ± 0%
Transformer/small/no_size_hint,_with_buffer-8     13.40Ki ± 0%
Transformer/small/with_size_hint,_with_buffer-8   13.39Ki ± 0%
Transformer/big/no_size_hint-8                    288.0Mi ± 0%
Transformer/big/no_size_hint,_with_buffer-8       1.390Gi ± 0%
Transformer/big/with_size_hint,_with_buffer-8     288.0Mi ± 0%
geomean                                           2.533Mi

                                                │    out     │
                                                │ allocs/op  │
Transformer/small/no_size_hint-8                  92.00 ± 0%
Transformer/small/no_size_hint,_with_buffer-8     92.00 ± 0%
Transformer/small/with_size_hint,_with_buffer-8   92.00 ± 0%
Transformer/big/no_size_hint-8                    546.5 ± 0%
Transformer/big/no_size_hint,_with_buffer-8       607.5 ± 0%
Transformer/big/with_size_hint,_with_buffer-8     545.5 ± 0%
geomean                                           228.1
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-30 12:13:04 +00:00
49ad985cad [#161] *: Do not use math/rand.Read()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-08 17:17:02 +03:00
c359a7465a [#64] transformer: Simplify interface
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-11 07:33:12 +00:00
ac95b87e7c [#101] Add Equals for Address
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-10 09:05:01 +00:00
aa8ffebc63 [#95] transformer: Set parent version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-23 13:48:02 +03:00
d4fe9a193d [#66] transformer: Accept constructor in NextTarget
The code of frostfs-node is not yet ready to reuse egress target for
multiple objects, let's postpone until #64.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 11:22:09 +03:00
c42a6119ff [#66] transformer: Extend tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 11:22:09 +03:00
29b188db57 [#52] Remove storage groups and audit
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 08:20:37 +00:00
38b03ff28b [#63] transformer: Resolve funlen linter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-25 13:29:42 +03:00
0fa23a9b14 [#63] transformer: Add context to methods
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-25 13:29:42 +03:00
708d933fe3 [#47] .golangci.yml: Unify with other FrostFS repos
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 08:59:19 +03:00
d589d51509 [#19] transformer: Fix dependencies
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-16 09:51:03 +03:00
25588ee3be [#19] transformer: Do not allocate intermeate slice for hashers
```
name                 old time/op    new time/op    delta
Transformer/small-8    73.7µs ±15%    72.4µs ±16%    ~     (p=0.604 n=10+9)
Transformer/big-8       1.36s ± 4%     1.36s ± 8%    ~     (p=0.579 n=10+10)

name                 old alloc/op   new alloc/op   delta
Transformer/small-8    7.67kB ± 0%    7.57kB ± 0%  -1.36%  (p=0.000 n=10+10)
Transformer/big-8      49.0kB ± 0%    48.3kB ± 0%  -1.48%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
Transformer/small-8       101 ± 0%        98 ± 0%  -2.97%  (p=0.000 n=10+10)
Transformer/big-8         609 ± 0%       591 ± 1%  -3.00%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
9407f30248 [#19] transformer: Optimize payload hashers a bit
```
name                 old time/op    new time/op    delta
Transformer/small-8    74.8µs ±11%    73.7µs ±15%    ~     (p=0.529 n=10+10)
Transformer/big-8       1.38s ±11%     1.36s ± 4%    ~     (p=0.796 n=10+10)

name                 old alloc/op   new alloc/op   delta
Transformer/small-8    7.69kB ± 0%    7.67kB ± 0%  -0.21%  (p=0.000 n=10+10)
Transformer/big-8      49.2kB ± 0%    49.0kB ± 0%  -0.48%  (p=0.004 n=10+10)

name                 old allocs/op  new allocs/op  delta
Transformer/small-8       102 ± 0%       101 ± 0%  -0.98%  (p=0.000 n=9+10)
Transformer/big-8         620 ± 1%       609 ± 0%  -1.66%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
94c0a607b5 [#19] transformer: Add a target which sends parts to a channel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
e45647de3c [#19] transformer: Do not reuse memory of sent objects
Slower, but more correct.
```
name                 old time/op    new time/op    delta
Transformer/small-8    72.4µs ± 8%    74.8µs ±11%     ~     (p=0.278 n=9+10)
Transformer/big-8       1.31s ± 8%     1.38s ±11%   +5.50%  (p=0.035 n=10+10)

name                 old alloc/op   new alloc/op   delta
Transformer/small-8    7.39kB ± 0%    7.69kB ± 0%   +4.04%  (p=0.000 n=10+10)
Transformer/big-8      46.9kB ± 0%    49.2kB ± 0%   +4.87%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
Transformer/small-8      94.6 ± 1%     102.0 ± 0%   +7.82%  (p=0.000 n=10+9)
Transformer/big-8         560 ± 0%       620 ± 1%  +10.66%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
611e20587b [#19] transformer/test: Check owner ID and payload hash for parts
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
eba6831125 [#19] transformer/test: Add helper functions
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
7e3810d654 [#19] transformer: Move EpochSource to other types
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
cc0fef2c55 [#19] transformer: Merge formatter and payload splitter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
b696d3c70e [#19] transformer: Make writeChunk non-recursive
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
1c94309d7a [#19] transformer: Simplify AccessIdentifiers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
f43f18ecda [#19] transformer: Cover with unit-tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
ac8442bf99 [#19] object: Move transformer implementation from node
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
b461aa64b8 [#23] pre-commit: Add golangci-lint hook
Also, fix minor issues. Skip deprecated warning for now.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-09 22:53:14 +03:00
94476f9055 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 15:47:21 +03:00
4ff9c00de3 [#4] Rename NeoFS mentions in comments and method names
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 13:40:50 +03:00
Pavel Karpy
a1748ae0e7 [#370] bearer, session: Clarify expiration epoch
The expiration epoch is the _last_ valid epoch for an entity. Also, clarify
the expiration epoch meaning for tombstones and regular objects.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-29 16:34:13 +03:00
Pavel Karpy
4c779423f5 Move to frostfs-sdk-go
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-14 09:59:29 +03:00
Denis Kirillov
d047289182 [#360] object: Add new package relations
Package relations provides feature to process inner object structure.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-14 17:46:17 +03:00
Denis Kirillov
74234623b2 [#215] object: Add well-known FilePath attribute
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-06 17:21:37 +04:00
Evgenii Stratonikov
84888854ab [#322] *: Go fmt -s
go1.19 rewrites comments to proper render them in docs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-24 18:58:59 +03:00
Leonard Lyubich
7d10b432d1 [#284] *: Return error from all ReadFromV2 methods
Return `error` from all `ReadFromV2` methods in order to support
backward compatibility if message will be extended with some formatted
field.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-07 14:20:18 +03:00
Leonard Lyubich
30d27c3050 [#289] oid: Add JSON encoding of Address type
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-04 14:58:00 +03:00
Leonard Lyubich
82d762f536 [#258] Fix string encoding of identifiers
Use `EncodeToString` method to get protocol string according to type
docs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-31 12:14:04 +03:00
Leonard Lyubich
6cb513c976 [#257] Upgrade NeoFS API Go module
New version contains fix for `object.GetRangeResponse` message type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-31 09:47:55 +03:00
Leonard Lyubich
3bbf7ee15d [#252] object/id: Fix typos in docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 12:41:35 +03:00