cb813e27a8
[ #293 ] object: Fix payload size limiter
...
* Sending empty chunks by `writeChunk` should not release new
objects as this doesn't change `payloadSizeLimiter` internal
state.
* This also fixes the bug with patcher when an offset of a patch
equals to `MaxSize` - `payloadSizeLimiter` releases object again
although state is the same. This led to error because EC-encoder
receieved empty payload and couldn't not process it.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-07 12:11:19 +00:00
6ce73790ea
[ #276 ] Merge repo with frostfs-api-go
...
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-10-22 14:05:12 +00:00
e580ee991d
[ #271 ] Drop handling of system attributes with NeoFS prefix
...
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-18 09:59:38 +00:00
6821fe6fb2
[ #271 ] object: Add UserAttributes method
...
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-18 09:59:38 +00:00
46ee543899
[ #265 ] go.mod: Use range over int
...
Since Go 1.22 a `for` statement with a `range` clause is able
to iterate through integer values from zero to an upper limit.
gopatch script:
@@
var i, e expression
@@
-for i := 0; i <= e - 1; i++ {
+for i := range e {
...
}
@@
var i, e expression
@@
-for i := 0; i <= e; i++ {
+for i := range e + 1 {
...
}
@@
var i, e expression
@@
-for i := 0; i < e; i++ {
+for i := range e {
...
}
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-04 12:37:46 +03:00
98aabc45a7
[ #252 ] patcher: Fix applying patch from the same offset
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-13 18:58:21 +03:00
361739e860
[ #247 ] object: Introduce patcher package
...
* Introduce `patcher` package that contains such interfaces to be
implemented:
- `PatchApplier` - the main patching engine that merges the stream
of patches and the stream of original object payload divided by
ranges. The merged streams result is output to `ChunkedObjectWriter`;
- `RangeProvider` - provides the original object payload by ranges;
- `HeaderProvider` - provides the original object header.
* Introduce `patcher` that implements `PatchApplier`;
* Cover all possible cases with unit-tests.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-05 12:32:14 +00:00
6dd500def9
[ #247 ] object: Introduce Patch type
...
* Make ToV2, FromV2 converters for Patch and PayloadPatch;
* Add unit-tests.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-05 12:32:14 +00:00
7e94a6adf2
[ #237 ] pool: Return creation epoch from object put
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-07-22 06:15:23 +00:00
ebd8fcd168
[ #224 ] object: Introduce parent attributes in EC-header
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 16:20:48 +03:00
3de256d05e
[ #223 ] object: Introduce new fields for ECHeader
...
* Introduce `parentSplitID`, `parentSplitParentID` fields
for `ECHeader`;
* Fix ECHeader's constructor;
* Fix `Split` and `Reconstruct`;
* Add unit-tests.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-17 15:16:28 +03:00
09b79d13f3
[ #223 ] object: Introduce ec_parent search filter
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-17 15:03:40 +03:00
20ab57bf7e
[ #214 ] object: Implement Get\Head requests for EC object
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-24 11:07:26 +03:00
ec0cb2169f
[ #211 ] object: Fix setIDWithSignature
...
* Calculate and set checksum before ID is calculated.
* Add header verification for parts in unit-test.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-29 13:48:04 +03:00
1af9b6d18b
[ #155 ] sdk-go: Add buffer support for payloadSizeLimiter
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-03-25 10:47:04 +03:00
bd2d350b09
[ #205 ] object: Initial EC implementation
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-22 10:14:12 +00:00
a86170f53a
[ #202 ] object: Reset marshal data on CutPayload
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-15 15:23:47 +03:00
56debcfa56
[ #190 ] sdk-go: Gofumpt fixes
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 19:21:20 +03:00
157a9930e8
[ #190 ] sdk-go: Pass user.ID by value
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 19:21:20 +03:00
665e5807bc
[ #188 ] transformer: Allow to provide size hint
...
For big objects with known size we can optimize allocation patterns
by providing size hint. As with any hint, it does not affect transformer
functionality: slices with capacity > MaxSize are never allocated.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/transformer
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ out │
│ sec/op │
Transformer/small/no_size_hint-8 65.44µ ± 3%
Transformer/small/no_size_hint,_with_buffer-8 64.24µ ± 5%
Transformer/small/with_size_hint,_with_buffer-8 58.70µ ± 5%
Transformer/big/no_size_hint-8 367.8m ± 3%
Transformer/big/no_size_hint,_with_buffer-8 562.7m ± 0%
Transformer/big/with_size_hint,_with_buffer-8 385.6m ± 7%
geomean 5.197m
│ out │
│ B/op │
Transformer/small/no_size_hint-8 13.40Ki ± 0%
Transformer/small/no_size_hint,_with_buffer-8 13.40Ki ± 0%
Transformer/small/with_size_hint,_with_buffer-8 13.39Ki ± 0%
Transformer/big/no_size_hint-8 288.0Mi ± 0%
Transformer/big/no_size_hint,_with_buffer-8 1.390Gi ± 0%
Transformer/big/with_size_hint,_with_buffer-8 288.0Mi ± 0%
geomean 2.533Mi
│ out │
│ allocs/op │
Transformer/small/no_size_hint-8 92.00 ± 0%
Transformer/small/no_size_hint,_with_buffer-8 92.00 ± 0%
Transformer/small/with_size_hint,_with_buffer-8 92.00 ± 0%
Transformer/big/no_size_hint-8 546.5 ± 0%
Transformer/big/no_size_hint,_with_buffer-8 607.5 ± 0%
Transformer/big/with_size_hint,_with_buffer-8 545.5 ± 0%
geomean 228.1
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-30 12:13:04 +00:00
49ad985cad
[ #161 ] *: Do not use math/rand.Read()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-08 17:17:02 +03:00
c359a7465a
[ #64 ] transformer: Simplify interface
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-11 07:33:12 +00:00
ac95b87e7c
[ #101 ] Add Equals for Address
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-10 09:05:01 +00:00
aa8ffebc63
[ #95 ] transformer: Set parent version
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-23 13:48:02 +03:00
d4fe9a193d
[ #66 ] transformer: Accept constructor in NextTarget
...
The code of frostfs-node is not yet ready to reuse egress target for
multiple objects, let's postpone until #64 .
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 11:22:09 +03:00
c42a6119ff
[ #66 ] transformer: Extend tests
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 11:22:09 +03:00
29b188db57
[ #52 ] Remove storage groups and audit
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 08:20:37 +00:00
38b03ff28b
[ #63 ] transformer: Resolve funlen linter
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-25 13:29:42 +03:00
0fa23a9b14
[ #63 ] transformer: Add context to methods
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-25 13:29:42 +03:00
708d933fe3
[ #47 ] .golangci.yml: Unify with other FrostFS repos
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 08:59:19 +03:00
d589d51509
[ #19 ] transformer: Fix dependencies
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-16 09:51:03 +03:00
25588ee3be
[ #19 ] transformer: Do not allocate intermeate slice for hashers
...
```
name old time/op new time/op delta
Transformer/small-8 73.7µs ±15% 72.4µs ±16% ~ (p=0.604 n=10+9)
Transformer/big-8 1.36s ± 4% 1.36s ± 8% ~ (p=0.579 n=10+10)
name old alloc/op new alloc/op delta
Transformer/small-8 7.67kB ± 0% 7.57kB ± 0% -1.36% (p=0.000 n=10+10)
Transformer/big-8 49.0kB ± 0% 48.3kB ± 0% -1.48% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Transformer/small-8 101 ± 0% 98 ± 0% -2.97% (p=0.000 n=10+10)
Transformer/big-8 609 ± 0% 591 ± 1% -3.00% (p=0.000 n=10+9)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
9407f30248
[ #19 ] transformer: Optimize payload hashers a bit
...
```
name old time/op new time/op delta
Transformer/small-8 74.8µs ±11% 73.7µs ±15% ~ (p=0.529 n=10+10)
Transformer/big-8 1.38s ±11% 1.36s ± 4% ~ (p=0.796 n=10+10)
name old alloc/op new alloc/op delta
Transformer/small-8 7.69kB ± 0% 7.67kB ± 0% -0.21% (p=0.000 n=10+10)
Transformer/big-8 49.2kB ± 0% 49.0kB ± 0% -0.48% (p=0.004 n=10+10)
name old allocs/op new allocs/op delta
Transformer/small-8 102 ± 0% 101 ± 0% -0.98% (p=0.000 n=9+10)
Transformer/big-8 620 ± 1% 609 ± 0% -1.66% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
94c0a607b5
[ #19 ] transformer: Add a target which sends parts to a channel
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
e45647de3c
[ #19 ] transformer: Do not reuse memory of sent objects
...
Slower, but more correct.
```
name old time/op new time/op delta
Transformer/small-8 72.4µs ± 8% 74.8µs ±11% ~ (p=0.278 n=9+10)
Transformer/big-8 1.31s ± 8% 1.38s ±11% +5.50% (p=0.035 n=10+10)
name old alloc/op new alloc/op delta
Transformer/small-8 7.39kB ± 0% 7.69kB ± 0% +4.04% (p=0.000 n=10+10)
Transformer/big-8 46.9kB ± 0% 49.2kB ± 0% +4.87% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Transformer/small-8 94.6 ± 1% 102.0 ± 0% +7.82% (p=0.000 n=10+9)
Transformer/big-8 560 ± 0% 620 ± 1% +10.66% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
611e20587b
[ #19 ] transformer/test: Check owner ID and payload hash for parts
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
eba6831125
[ #19 ] transformer/test: Add helper functions
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
7e3810d654
[ #19 ] transformer: Move EpochSource to other types
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
cc0fef2c55
[ #19 ] transformer: Merge formatter and payload splitter
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
b696d3c70e
[ #19 ] transformer: Make writeChunk non-recursive
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
1c94309d7a
[ #19 ] transformer: Simplify AccessIdentifiers
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
f43f18ecda
[ #19 ] transformer: Cover with unit-tests
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
ac8442bf99
[ #19 ] object: Move transformer implementation from node
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-15 05:39:09 +00:00
b461aa64b8
[ #23 ] pre-commit: Add golangci-lint hook
...
Also, fix minor issues. Skip deprecated warning for now.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-09 22:53:14 +03:00
94476f9055
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 15:47:21 +03:00
4ff9c00de3
[ #4 ] Rename NeoFS mentions in comments and method names
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 13:40:50 +03:00
Pavel Karpy
a1748ae0e7
[ #370 ] bearer, session: Clarify expiration epoch
...
The expiration epoch is the _last_ valid epoch for an entity. Also, clarify
the expiration epoch meaning for tombstones and regular objects.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-29 16:34:13 +03:00
Pavel Karpy
4c779423f5
Move to frostfs-sdk-go
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-14 09:59:29 +03:00
Denis Kirillov
d047289182
[ #360 ] object: Add new package relations
...
Package relations provides feature to process inner object structure.
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-14 17:46:17 +03:00
Denis Kirillov
74234623b2
[ #215 ] object: Add well-known FilePath attribute
...
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-06 17:21:37 +04:00
