realloc/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Releases Packages Activity
5375 commits 1 branch 26 tags 15 MiB
Commit graph

1270 commits

Author SHA1 Message Date
Dmitrii Stepanov
07a660fbc4
[#1677] writecache: Add QoS limiter usage 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-03-14 16:23:33 +03:00
Evgenii Stratonikov
460e5cbccf [#1671] Use slices.Delete() where possible 
gopatch is missing for this one, because
https://github.com/uber-go/gopatch/issues/179

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-03-13 08:12:20 +00:00
Evgenii Stratonikov
155d3ddb6e [#1671] Use min builtin where possible 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-03-13 08:12:20 +00:00
Evgenii Stratonikov
40536d8a06 [#1671] Use fmt.Appendf where warranted 
Fix gopls warnings:
```
cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf
````

gopatch:
```
@@
var f expression
@@
-[]byte(fmt.Sprintf(f, ...))
+fmt.Appendf(nil, f, ...)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-03-13 08:12:20 +00:00
Evgenii Stratonikov
bcc84c85a0 [#1671] Replace interface{} with any 
gopatch:
```
@@
@@
-interface{}
+any
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-03-13 08:12:20 +00:00
Dmitrii Stepanov
2005fdda09
[#1667] shard: Drop shard pool 
After adding an ops limiter, shard's `put` pool is redundant.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-03-11 13:59:51 +03:00
Dmitrii Stepanov
3727d60331 [#1653] qos: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-03-11 10:57:47 +00:00
Dmitrii Stepanov
8643e0abc5
[#1668] writecache: Use object size to check free space 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-03-10 17:52:57 +03:00
Ekaterina Lebedeva
9e31cb249f [#1635] control: Add method to search shards by object 
Added method `ListShardsForObject` to ControlService and to
StorageEngine. It returns information about shards storing
object on the node.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-03-07 14:32:01 +03:00
Dmitrii Stepanov
4685afb1dc
[#1636] engine: Validate limiter release in unit tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:30 +03:00
Dmitrii Stepanov
eb8b9b2b3b
[#1636] blobovniczatree: Validate limiter release in rebuild unit tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:30 +03:00
Dmitrii Stepanov
6c6e463b73
[#1636] shard: Change ops limiter on shard reload 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:29 +03:00
Dmitrii Stepanov
c2d855aedd
[#1636] qos: Return Resource Exhausted error 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:28 +03:00
Dmitrii Stepanov
b9360be1dc
[#1636] blobovniczatree: Use RebuildLimiter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:27 +03:00
Dmitrii Stepanov
ceff5e1f6a
[#1636] storage: Refactor shard rebuild 
Drop redundant interfaces.
Rename fields.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:27 +03:00
Dmitrii Stepanov
e0dc3c3d0c
[#1636] shard: Add limiter usage 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:26 +03:00
Dmitrii Stepanov
0991077cb3 [#1657] engine: Fix data race in evacuation tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 14:14:12 +00:00
Aleksey Savchuk
02f3a7f65c
[#1648] writecache: Fix race condition when reporting cache size metrics 
There is a race condition when multiple cache operation try to report
the cache size metrics simultaneously. Consider the following example:
- the initial total size of objects stored in the cache size is 2
- worker X deletes an object and reads the cache size, which is 1
- worker Y deletes an object and reads the cache size, which is 0
- worker Y reports the cache size it learnt, which is 0
- worker X reports the cache size it learnt, which is 1

As a result, the observed cache size is 1 (i. e. one object remains
in the cache), which is incorrect because the actual cache size is 0.

To fix this, let's report the metrics periodically in the flush loop.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-19 17:05:40 +03:00
Dmitrii Stepanov
5d79abe523
[#1608] shard: Add IO tag for rebuild 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:07 +03:00
Dmitrii Stepanov
92450a76ba
[#1608] shard: Add IO tag to GC 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:06 +03:00
Dmitrii Stepanov
abba5b2089
[#1608] writecache: Add IO tag to flush worker 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:06 +03:00
Dmitrii Stepanov
f6b3f79e89
[#1608] qos: Add qos service to adjust incoming IO tags 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:03 +03:00
Alexander Chuprov
9b113c3156
[#1613] morph: Add tracing for morph queries to neo-go 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-05 16:38:20 +03:00
Dmitrii Stepanov
4de5fca547
[#1624] blobovniczatree: Return source error from Init() 
As it was before: when the database was opened, an error returned,
but along with the original error, the `context cancelled`` error returned,
because `iterateIncompletedRebuildDBPaths` method has `ctx.Done()` check
and egCtx passed to `iterateIncompletedRebuildDBPaths` method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-03 17:36:21 +03:00
Dmitrii Stepanov
e1a984e9d8
[#1620] logs: Drop redundant trace_id fields 
`trace_id` is taken from context.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-01-29 16:13:51 +03:00
Evgenii Stratonikov
c98357606b
[#1606] Use slices.Clone()/bytes.Clone() where possible 
gopatch:
```
@@
var from, to expression
@@
+import "bytes"
-to := make([]byte, len(from))
-copy(to, from)
+to := bytes.Clone(from)

@@
var from, to expression
@@
+import "bytes"
-to = make([]byte, len(from))
-copy(to, from)
+to = bytes.Clone(from)

@@
var from, to, typ expression
@@
+import "slices"
-to := make([]typ, len(from))
-copy(to, from)
+to := slices.Clone(from)

@@
var from, to, typ expression
@@
+import "slices"
-to = make([]typ, len(from))
-copy(to, from)
+to = slices.Clone(from)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-17 14:50:14 +03:00
Evgenii Stratonikov
05fd999162
[#1600] fstree: Handle incomplete writes 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-14 14:52:35 +03:00
Dmitrii Stepanov
eff95bd632
[#1598] engine: Drop unnecessary result structs 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-01-14 11:15:21 +03:00
Dmitrii Stepanov
fb928616cc
[#1598] golangci: Enable unparam linter 
To drop unnecessary parameters and return values.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-01-14 09:06:47 +03:00
Dmitrii Stepanov
4d5ae59a52
[#1598] golangci: Enable unconvert linters 
To drop unnecessary conversions.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-01-14 09:06:40 +03:00
Anton Nikiforov
6c51f48aab [#1596] metrics: Create public aliases for internal engine metrics 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2025-01-13 10:05:01 +00:00
Evgenii Stratonikov
3900b92927
Revert "[#1492] metabase: Ensure Unmarshal() is called on a cloned slice" 
This reverts commit 8ed7a676d5.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-09 14:34:20 +03:00
Aleksey Savchuk
fa08bfa553
[#1583] metabase/test: Update TestLisObjectsWithCursor 
Update this test following recent changes to ensure
that `(*DB).ListWithCursor` skips expired objects.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-26 14:39:50 +03:00
Aleksey Savchuk
0da998ef50
[#1583] metabase: Skip expired objects in ListWithCursor 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-26 14:39:49 +03:00
Evgenii Stratonikov
226dd25dd0 [#1568] pilorama: Replace "containerID" with "container ID" in the error message 
It is "container ID" in every other place.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-12-18 15:52:26 +00:00
Evgenii Stratonikov
bd0197eaa8 [#1568] storage: Remove "could not/can't/failed to" from error messages 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-12-18 15:52:26 +00:00
Aleksey Savchuk
f0c43c8d80
[#1502] Use zap.Error for logging errors 
Use `zap.Error` instead of `zap.String` for logging errors: change all expressions like
`zap.String("error", err.Error())` or `zap.String("err", err.Error())` to `zap.Error(err)`.
Leave similar expressions with other messages unchanged, for example,
`zap.String("last_error", lastErr.Error())` or `zap.String("reason", ctx.Err().Error())`.

This change was made by applying the following patch:
```diff
@@
var err expression
@@
-zap.String("error", err.Error())
+zap.Error(err)

@@
var err expression
@@
-zap.String("err", err.Error())
+zap.Error(err)
```

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-16 11:13:42 +03:00
Aleksey Savchuk
8ba9f31fca
[#1510] metabase/test: Fix BenchmarkListWithCursor 
- Fix misplaced `(*DB).Close` (broken after 47dcfa20f3)
- Use `errors.Is` for error checking (broken after fcdbf5e509)

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-13 13:19:15 +03:00
Aleksey Savchuk
2af3409d39
[#1510] metabase/test: Fix BenchmarkGet 
Fix misplaced `(*DB).Close` (broken after 47dcfa20f3)

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-13 13:18:43 +03:00
Aleksey Savchuk
3821645085
[#1555] engine: Refactor (*StorageEngine).GetLocks 
Refactored after renaming the method to replace the confusing `locked`
variable with `locks`.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-11 15:06:38 +03:00
Aleksey Savchuk
72470d6b48
[#1555] local_object_storage: Rename method GetLocked -> GetLocks 
Renamed to better reflect the method's purpose of returning locks
for the specified object.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-11 15:06:37 +03:00
Dmitrii Stepanov
1f6cf57e30 [#1548] metabase: Check if EC parent is removed or expired 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-12-11 07:26:33 +00:00
Dmitrii Stepanov
386a12eea4 [#1548] engine: Rename parent -> ecParent 
Parent could mean split parent or EC parent. In this case it is EC parent only.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-12-11 07:26:33 +00:00
Dmitrii Stepanov
41da27dad5
[#1549] engine: Drop Async flag from evacuation parameters 
Now it is only async evacuation.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-12-10 17:00:00 +03:00
Aleksey Savchuk
281d65435e
[#1450] engine: Group object by shard before Inhume 
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine
cpu: 12th Gen Intel(R) Core(TM) i5-1235U
                                 │   old.txt    │              new.txt                │
                                 │    sec/op    │   sec/op     vs base                │
InhumeMultipart/objects=1-12        11.42m ± 1%   10.71m ± 0%   -6.27% (p=0.000 n=10)
InhumeMultipart/objects=10-12       113.5m ± 0%   100.9m ± 3%  -11.08% (p=0.000 n=10)
InhumeMultipart/objects=100-12     1135.4m ± 1%   681.3m ± 2%  -40.00% (p=0.000 n=10)
InhumeMultipart/objects=1000-12     11.358 ± 0%    1.089 ± 1%  -90.41% (p=0.000 n=10)
InhumeMultipart/objects=10000-12   113.251 ± 0%    1.645 ± 1%  -98.55% (p=0.000 n=10)
geomean                              1.136        265.5m       -76.63%
```

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-04 10:09:00 +03:00
Aleksey Savchuk
b348b20289
[#1450] engine: Add benchmark for Inhume operation 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-04 10:08:34 +03:00
Aleksey Savchuk
748edd1999
[#1450] engine: Return shard-level error if object is expired on inhume 
Since we have errors defined on the shard-level, it looks strage that we
check an error againt the shard-level error `ErrLockObjectRemoval`, but
then return the metabase-level error. Let's return the same shard-level
error instead.

Since we have errors defined on the shard-level

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-04 10:06:57 +03:00
Aleksey Savchuk
432042c534
[#1527] engine: Add tests for handling expired objects on inhume and lock 
Currently, it's allowed to inhume or lock an expired object.
Consider the following scenario:

1) An user inhumes or locks an object
2) The object expires
3) GC hasn't yet deleted the object
4) The node loses the associated tombstone or lock
5) Another node replicates tombstone or lock to the first node

In this case, the second node succeeds, which is the desired behavior.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-03 12:29:45 +03:00
Aleksey Savchuk
9cabca9dfe
[#1527] engine/test: Move default metabase options to separate function 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-02 16:13:37 +03:00
Aleksey Savchuk
60feed3b5f
[#1527] engine/test: Allow to specify current epoch in epochState 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2024-12-02 15:37:25 +03:00