a11b2d27e4
[ #1642 ] tree: Introduce Cursor type
...
* Use `Cursor` as parameter for `TreeSortedByFilename`
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a405fb1f39
[ #1683 ] metabase: Check object status once in Select()
...
objectStatus() is called twice for the same object:
First, in selectObject() to filter removed objects.
Then, again, in getObjectForSlowFilters() via db.get().
The second call will return the same result, so remove useless branch.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ status │
│ sec/op │ sec/op vs base │
Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10)
Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10)
Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10)
Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10)
geomean 5.045m 3.865m -23.39%
│ old │ status │
│ B/op │ B/op vs base │
Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10)
geomean 2.683Mi 2.248Mi -16.21%
│ old │ status │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10)
geomean 68.78k 55.77k -18.90%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
a7319bc979
[ #1683 ] metabase/test: Report allocs in benchmarkSelect()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
07a660fbc4
[ #1677 ] writecache: Add QoS limiter usage
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 16:23:33 +03:00
460e5cbccf
[ #1671 ] Use slices.Delete() where possible
...
gopatch is missing for this one, because
https://github.com/uber-go/gopatch/issues/179
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
155d3ddb6e
[ #1671 ] Use min builtin where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
40536d8a06
[ #1671 ] Use fmt.Appendf where warranted
...
Fix gopls warnings:
```
cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf
````
gopatch:
```
@@
var f expression
@@
-[]byte(fmt.Sprintf(f, ...))
+fmt.Appendf(nil, f, ...)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
bcc84c85a0
[ #1671 ] Replace interface{} with any
...
gopatch:
```
@@
@@
-interface{}
+any
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
2005fdda09
[ #1667 ] shard: Drop shard pool
...
After adding an ops limiter, shard's `put` pool is redundant.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 13:59:51 +03:00
3727d60331
[ #1653 ] qos: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
8643e0abc5
[ #1668 ] writecache: Use object size to check free space
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-10 17:52:57 +03:00
9e31cb249f
[ #1635 ] control: Add method to search shards by object
...
Added method `ListShardsForObject` to ControlService and to
StorageEngine. It returns information about shards storing
object on the node.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
4685afb1dc
[ #1636 ] engine: Validate limiter release in unit tests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:30 +03:00
eb8b9b2b3b
[ #1636 ] blobovniczatree: Validate limiter release in rebuild unit tests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:30 +03:00
6c6e463b73
[ #1636 ] shard: Change ops limiter on shard reload
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:29 +03:00
c2d855aedd
[ #1636 ] qos: Return Resource Exhausted error
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:28 +03:00
b9360be1dc
[ #1636 ] blobovniczatree: Use RebuildLimiter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:27 +03:00
ceff5e1f6a
[ #1636 ] storage: Refactor shard rebuild
...
Drop redundant interfaces.
Rename fields.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:27 +03:00
e0dc3c3d0c
[ #1636 ] shard: Add limiter usage
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:26 +03:00
0991077cb3
[ #1657 ] engine: Fix data race in evacuation tests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 14:14:12 +00:00
02f3a7f65c
[ #1648 ] writecache: Fix race condition when reporting cache size metrics
...
There is a race condition when multiple cache operation try to report
the cache size metrics simultaneously. Consider the following example:
- the initial total size of objects stored in the cache size is 2
- worker X deletes an object and reads the cache size, which is 1
- worker Y deletes an object and reads the cache size, which is 0
- worker Y reports the cache size it learnt, which is 0
- worker X reports the cache size it learnt, which is 1
As a result, the observed cache size is 1 (i. e. one object remains
in the cache), which is incorrect because the actual cache size is 0.
To fix this, let's report the metrics periodically in the flush loop.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-19 17:05:40 +03:00
5d79abe523
[ #1608 ] shard: Add IO tag for rebuild
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:07 +03:00
92450a76ba
[ #1608 ] shard: Add IO tag to GC
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:06 +03:00
abba5b2089
[ #1608 ] writecache: Add IO tag to flush worker
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:06 +03:00
f6b3f79e89
[ #1608 ] qos: Add qos service to adjust incoming IO tags
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:03 +03:00
9b113c3156
[ #1613 ] morph: Add tracing for morph queries to neo-go
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-02-05 16:38:20 +03:00
4de5fca547
[ #1624 ] blobovniczatree: Return source error from Init()
...
As it was before: when the database was opened, an error returned,
but along with the original error, the `context cancelled`` error returned,
because `iterateIncompletedRebuildDBPaths` method has `ctx.Done()` check
and egCtx passed to `iterateIncompletedRebuildDBPaths` method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-03 17:36:21 +03:00
e1a984e9d8
[ #1620 ] logs: Drop redundant trace_id fields
...
`trace_id` is taken from context.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-01-29 16:13:51 +03:00
c98357606b
[ #1606 ] Use slices.Clone()/bytes.Clone() where possible
...
gopatch:
```
@@
var from, to expression
@@
+import "bytes"
-to := make([]byte, len(from))
-copy(to, from)
+to := bytes.Clone(from)
@@
var from, to expression
@@
+import "bytes"
-to = make([]byte, len(from))
-copy(to, from)
+to = bytes.Clone(from)
@@
var from, to, typ expression
@@
+import "slices"
-to := make([]typ, len(from))
-copy(to, from)
+to := slices.Clone(from)
@@
var from, to, typ expression
@@
+import "slices"
-to = make([]typ, len(from))
-copy(to, from)
+to = slices.Clone(from)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-01-17 14:50:14 +03:00
05fd999162
[ #1600 ] fstree: Handle incomplete writes
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-01-14 14:52:35 +03:00
eff95bd632
[ #1598 ] engine: Drop unnecessary result structs
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-01-14 11:15:21 +03:00
fb928616cc
[ #1598 ] golangci: Enable unparam linter
...
To drop unnecessary parameters and return values.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-01-14 09:06:47 +03:00
4d5ae59a52
[ #1598 ] golangci: Enable unconvert linters
...
To drop unnecessary conversions.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-01-14 09:06:40 +03:00
6c51f48aab
[ #1596 ] metrics: Create public aliases for internal engine metrics
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-01-13 10:05:01 +00:00
3900b92927
Revert "[ #1492 ] metabase: Ensure Unmarshal() is called on a cloned slice"
...
This reverts commit 8ed7a676d5
2025-01-09 14:34:20 +03:00
fa08bfa553
[ #1583 ] metabase/test: Update TestLisObjectsWithCursor
...
Update this test following recent changes to ensure
that `(*DB).ListWithCursor` skips expired objects.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-26 14:39:50 +03:00
0da998ef50
[ #1583 ] metabase: Skip expired objects in ListWithCursor
...
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-26 14:39:49 +03:00
226dd25dd0
[ #1568 ] pilorama: Replace "containerID" with "container ID" in the error message
...
It is "container ID" in every other place.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-18 15:52:26 +00:00
bd0197eaa8
[ #1568 ] storage: Remove "could not/can't/failed to" from error messages
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-18 15:52:26 +00:00
f0c43c8d80
[ #1502 ] Use zap.Error for logging errors
...
Use `zap.Error` instead of `zap.String` for logging errors: change all expressions like
`zap.String("error", err.Error())` or `zap.String("err", err.Error())` to `zap.Error(err)`.
Leave similar expressions with other messages unchanged, for example,
`zap.String("last_error", lastErr.Error())` or `zap.String("reason", ctx.Err().Error())`.
This change was made by applying the following patch:
```diff
@@
var err expression
@@
-zap.String("error", err.Error())
+zap.Error(err)
@@
var err expression
@@
-zap.String("err", err.Error())
+zap.Error(err)
```
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-16 11:13:42 +03:00
8ba9f31fca
[ #1510 ] metabase/test: Fix BenchmarkListWithCursor
...
- Fix misplaced `(*DB).Close` (broken after 47dcfa20f3fcdbf5e509
2024-12-13 13:19:15 +03:00
2af3409d39
[ #1510 ] metabase/test: Fix BenchmarkGet
...
Fix misplaced `(*DB).Close` (broken after 47dcfa20f3
2024-12-13 13:18:43 +03:00
3821645085
[ #1555 ] engine: Refactor (*StorageEngine).GetLocks
...
Refactored after renaming the method to replace the confusing `locked`
variable with `locks`.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-11 15:06:38 +03:00
72470d6b48
[ #1555 ] local_object_storage: Rename method GetLocked -> GetLocks
...
Renamed to better reflect the method's purpose of returning locks
for the specified object.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-11 15:06:37 +03:00
1f6cf57e30
[ #1548 ] metabase: Check if EC parent is removed or expired
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-11 07:26:33 +00:00
386a12eea4
[ #1548 ] engine: Rename parent -> ecParent
...
Parent could mean split parent or EC parent. In this case it is EC parent only.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-11 07:26:33 +00:00
41da27dad5
[ #1549 ] engine: Drop Async flag from evacuation parameters
...
Now it is only async evacuation.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-10 17:00:00 +03:00
281d65435e
[ #1450 ] engine: Group object by shard before Inhume
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine
cpu: 12th Gen Intel(R) Core(TM) i5-1235U
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
InhumeMultipart/objects=1-12 11.42m ± 1% 10.71m ± 0% -6.27% (p=0.000 n=10)
InhumeMultipart/objects=10-12 113.5m ± 0% 100.9m ± 3% -11.08% (p=0.000 n=10)
InhumeMultipart/objects=100-12 1135.4m ± 1% 681.3m ± 2% -40.00% (p=0.000 n=10)
InhumeMultipart/objects=1000-12 11.358 ± 0% 1.089 ± 1% -90.41% (p=0.000 n=10)
InhumeMultipart/objects=10000-12 113.251 ± 0% 1.645 ± 1% -98.55% (p=0.000 n=10)
geomean 1.136 265.5m -76.63%
```
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-04 10:09:00 +03:00
b348b20289
[ #1450 ] engine: Add benchmark for Inhume operation
...
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-04 10:08:34 +03:00
748edd1999
[ #1450 ] engine: Return shard-level error if object is expired on inhume
...
Since we have errors defined on the shard-level, it looks strage that we
check an error againt the shard-level error `ErrLockObjectRemoval`, but
then return the metabase-level error. Let's return the same shard-level
error instead.
Since we have errors defined on the shard-level
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-04 10:06:57 +03:00
