1275 commits

Author	SHA1	Message	Date
Airat Arifullin	39f549a7ab	[#1642] tree: Intoduce a helper LastChild ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	760b6a44ea	[#1642] tree: Fix sorted getSubtree for multiversion filenames ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	a11b2d27e4	[#1642] tree: Introduce Cursor type ... * Use `Cursor` as parameter for `TreeSortedByFilename` Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Evgenii Stratonikov	a405fb1f39	[#1683] metabase: Check object status once in Select() ... objectStatus() is called twice for the same object: First, in selectObject() to filter removed objects. Then, again, in getObjectForSlowFilters() via db.get(). The second call will return the same result, so remove useless branch. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ status │ │ sec/op │ sec/op vs base │ Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10) Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10) Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10) Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10) geomean 5.045m 3.865m -23.39% │ old │ status │ │ B/op │ B/op vs base │ Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10) geomean 2.683Mi 2.248Mi -16.21% │ old │ status │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10) geomean 68.78k 55.77k -18.90% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-18 11:48:51 +00:00
Evgenii Stratonikov	a7319bc979	[#1683] metabase/test: Report allocs in benchmarkSelect() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-18 11:48:51 +00:00
Dmitrii Stepanov	07a660fbc4	[#1677] writecache: Add QoS limiter usage ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-14 16:23:33 +03:00
Evgenii Stratonikov	460e5cbccf	[#1671] Use slices.Delete() where possible ... gopatch is missing for this one, because https://github.com/uber-go/gopatch/issues/179 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	155d3ddb6e	[#1671] Use min builtin where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	40536d8a06	[#1671] Use fmt.Appendf where warranted ... Fix gopls warnings: ``` cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf ```` gopatch: ``` @@ var f expression @@ -[]byte(fmt.Sprintf(f, ...)) +fmt.Appendf(nil, f, ...) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	bcc84c85a0	[#1671] Replace interface{} with any ... gopatch: ``` @@ @@ -interface{} +any ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Dmitrii Stepanov	2005fdda09	[#1667] shard: Drop shard pool ... After adding an ops limiter, shard's `put` pool is redundant. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 13:59:51 +03:00
Dmitrii Stepanov	3727d60331	[#1653] qos: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	8643e0abc5	[#1668] writecache: Use object size to check free space ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-10 17:52:57 +03:00
Ekaterina Lebedeva	9e31cb249f	[#1635] control: Add method to search shards by object ... Added method `ListShardsForObject` to ControlService and to StorageEngine. It returns information about shards storing object on the node. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 14:32:01 +03:00
Dmitrii Stepanov	4685afb1dc	[#1636] engine: Validate limiter release in unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:30 +03:00
Dmitrii Stepanov	eb8b9b2b3b	[#1636] blobovniczatree: Validate limiter release in rebuild unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:30 +03:00
Dmitrii Stepanov	6c6e463b73	[#1636] shard: Change ops limiter on shard reload ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:29 +03:00
Dmitrii Stepanov	c2d855aedd	[#1636] qos: Return Resource Exhausted error ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:28 +03:00
Dmitrii Stepanov	b9360be1dc	[#1636] blobovniczatree: Use RebuildLimiter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:27 +03:00
Dmitrii Stepanov	ceff5e1f6a	[#1636] storage: Refactor shard rebuild ... Drop redundant interfaces. Rename fields. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:27 +03:00
Dmitrii Stepanov	e0dc3c3d0c	[#1636] shard: Add limiter usage ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:26 +03:00
Dmitrii Stepanov	0991077cb3	[#1657] engine: Fix data race in evacuation tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 14:14:12 +00:00
Aleksey Savchuk	02f3a7f65c	[#1648] writecache: Fix race condition when reporting cache size metrics ... There is a race condition when multiple cache operation try to report the cache size metrics simultaneously. Consider the following example: - the initial total size of objects stored in the cache size is 2 - worker X deletes an object and reads the cache size, which is 1 - worker Y deletes an object and reads the cache size, which is 0 - worker Y reports the cache size it learnt, which is 0 - worker X reports the cache size it learnt, which is 1 As a result, the observed cache size is 1 (i. e. one object remains in the cache), which is incorrect because the actual cache size is 0. To fix this, let's report the metrics periodically in the flush loop. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2025-02-19 17:05:40 +03:00
Dmitrii Stepanov	5d79abe523	[#1608] shard: Add IO tag for rebuild ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-07 15:43:07 +03:00
Dmitrii Stepanov	92450a76ba	[#1608] shard: Add IO tag to GC ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-07 15:43:06 +03:00
Dmitrii Stepanov	abba5b2089	[#1608] writecache: Add IO tag to flush worker ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-07 15:43:06 +03:00
Dmitrii Stepanov	f6b3f79e89	[#1608] qos: Add qos service to adjust incoming IO tags ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-07 15:43:03 +03:00
Alexander Chuprov	9b113c3156	[#1613] morph: Add tracing for morph queries to neo-go ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2025-02-05 16:38:20 +03:00
Dmitrii Stepanov	4de5fca547	[#1624] blobovniczatree: Return source error from Init() ... As it was before: when the database was opened, an error returned, but along with the original error, the `context cancelled`` error returned, because `iterateIncompletedRebuildDBPaths` method has `ctx.Done()` check and egCtx passed to `iterateIncompletedRebuildDBPaths` method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-03 17:36:21 +03:00
Dmitrii Stepanov	e1a984e9d8	[#1620] logs: Drop redundant trace_id fields ... `trace_id` is taken from context. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-01-29 16:13:51 +03:00
Evgenii Stratonikov	c98357606b	[#1606] Use slices.Clone()/bytes.Clone() where possible ... gopatch: ``` @@ var from, to expression @@ +import "bytes" -to := make([]byte, len(from)) -copy(to, from) +to := bytes.Clone(from) @@ var from, to expression @@ +import "bytes" -to = make([]byte, len(from)) -copy(to, from) +to = bytes.Clone(from) @@ var from, to, typ expression @@ +import "slices" -to := make([]typ, len(from)) -copy(to, from) +to := slices.Clone(from) @@ var from, to, typ expression @@ +import "slices" -to = make([]typ, len(from)) -copy(to, from) +to = slices.Clone(from) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-01-17 14:50:14 +03:00
Evgenii Stratonikov	05fd999162	[#1600] fstree: Handle incomplete writes ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-01-14 14:52:35 +03:00
Dmitrii Stepanov	eff95bd632	[#1598] engine: Drop unnecessary result structs ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-01-14 11:15:21 +03:00
Dmitrii Stepanov	fb928616cc	[#1598] golangci: Enable unparam linter ... To drop unnecessary parameters and return values. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-01-14 09:06:47 +03:00
Dmitrii Stepanov	4d5ae59a52	[#1598] golangci: Enable unconvert linters ... To drop unnecessary conversions. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-01-14 09:06:40 +03:00
Anton Nikiforov	6c51f48aab	[#1596] metrics: Create public aliases for internal engine metrics ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2025-01-13 10:05:01 +00:00
Evgenii Stratonikov	3900b92927	Revert "[#1492] metabase: Ensure Unmarshal() is called on a cloned slice" ... This reverts commit 8ed7a676d5. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-01-09 14:34:20 +03:00
Aleksey Savchuk	fa08bfa553	[#1583] metabase/test: Update TestLisObjectsWithCursor ... Update this test following recent changes to ensure that `(*DB).ListWithCursor` skips expired objects. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-26 14:39:50 +03:00
Aleksey Savchuk	0da998ef50	[#1583] metabase: Skip expired objects in ListWithCursor ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-26 14:39:49 +03:00
Evgenii Stratonikov	226dd25dd0	[#1568] pilorama: Replace "containerID" with "container ID" in the error message ... It is "container ID" in every other place. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-12-18 15:52:26 +00:00
Evgenii Stratonikov	bd0197eaa8	[#1568] storage: Remove "could not/can't/failed to" from error messages ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-12-18 15:52:26 +00:00
Aleksey Savchuk	f0c43c8d80	[#1502] Use zap.Error for logging errors ... Use `zap.Error` instead of `zap.String` for logging errors: change all expressions like `zap.String("error", err.Error())` or `zap.String("err", err.Error())` to `zap.Error(err)`. Leave similar expressions with other messages unchanged, for example, `zap.String("last_error", lastErr.Error())` or `zap.String("reason", ctx.Err().Error())`. This change was made by applying the following patch: ```diff @@ var err expression @@ -zap.String("error", err.Error()) +zap.Error(err) @@ var err expression @@ -zap.String("err", err.Error()) +zap.Error(err) ``` Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-16 11:13:42 +03:00
Aleksey Savchuk	8ba9f31fca	[#1510] metabase/test: Fix BenchmarkListWithCursor ... - Fix misplaced `(*DB).Close` (broken after 47dcfa20f3) - Use `errors.Is` for error checking (broken after fcdbf5e509) Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-13 13:19:15 +03:00
Aleksey Savchuk	2af3409d39	[#1510] metabase/test: Fix BenchmarkGet ... Fix misplaced `(*DB).Close` (broken after 47dcfa20f3) Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-13 13:18:43 +03:00
Aleksey Savchuk	3821645085	[#1555] engine: Refactor (*StorageEngine).GetLocks ... Refactored after renaming the method to replace the confusing `locked` variable with `locks`. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-11 15:06:38 +03:00
Aleksey Savchuk	72470d6b48	[#1555] local_object_storage: Rename method GetLocked -> GetLocks ... Renamed to better reflect the method's purpose of returning locks for the specified object. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-11 15:06:37 +03:00
Dmitrii Stepanov	1f6cf57e30	[#1548] metabase: Check if EC parent is removed or expired ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-12-11 07:26:33 +00:00
Dmitrii Stepanov	386a12eea4	[#1548] engine: Rename parent -> ecParent ... Parent could mean split parent or EC parent. In this case it is EC parent only. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-12-11 07:26:33 +00:00
Dmitrii Stepanov	41da27dad5	[#1549] engine: Drop Async flag from evacuation parameters ... Now it is only async evacuation. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-12-10 17:00:00 +03:00
Aleksey Savchuk	281d65435e	[#1450] engine: Group object by shard before Inhume ... ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine cpu: 12th Gen Intel(R) Core(TM) i5-1235U │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ InhumeMultipart/objects=1-12 11.42m ± 1% 10.71m ± 0% -6.27% (p=0.000 n=10) InhumeMultipart/objects=10-12 113.5m ± 0% 100.9m ± 3% -11.08% (p=0.000 n=10) InhumeMultipart/objects=100-12 1135.4m ± 1% 681.3m ± 2% -40.00% (p=0.000 n=10) InhumeMultipart/objects=1000-12 11.358 ± 0% 1.089 ± 1% -90.41% (p=0.000 n=10) InhumeMultipart/objects=10000-12 113.251 ± 0% 1.645 ± 1% -98.55% (p=0.000 n=10) geomean 1.136 265.5m -76.63% ``` Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-04 10:09:00 +03:00