3542 commits

Author	SHA1	Message	Date
Dmitrii Stepanov	5385f9994f	[#1695] mod: Bump frostfs-observability version ... Change-Id: Id362b71f743ff70c8cd374030c9fa67e2566022f Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-21 13:28:02 +03:00
Dmitrii Stepanov	eea46a599d	[#1695] qos: Add treesync tag ... Tree sync is too much different from GC and rebuild to use the same tag for GC and tree sync. Change-Id: Ib44d5fa9a88daff507d759d0b0410cc9272e236f Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-21 13:28:02 +03:00
Anton Nikiforov	049a650b89	[#1619] logger: Simplify logger config reloading ... Change-Id: Ide892b250304b8cdb6c279f5f728c3b35f05df54 Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2025-03-21 09:03:54 +00:00
Evgenii Stratonikov	3f4717a37f	[#1692] metabase: Do not allocate map in cache unless needed ... Change-Id: I8b1015a8c7c3df4153a08fdb788117d9f0d6c333 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-21 08:56:32 +00:00
Evgenii Stratonikov	60cea8c714	[#1692] metabase/test: Fix end of iteration error check ... This is not good: ``` BenchmarkListWithCursor/1_item-8 --- FAIL: BenchmarkListWithCursor/1_item-8 list_test.go:63: error: end of object listing ``` Change-Id: I61b70937ce30fefaf16ebeb0cdb51bdd39096061 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-21 08:56:18 +00:00
Evgenii Stratonikov	e8801dbf49	[#1691] metabase: Move cheaper conditions to the front in ListWithCursor() ... `objectLocked` call is expensive, it does IO. We may omit it if object is not expired. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:36 +00:00
Evgenii Stratonikov	eb9df85b98	[#1685] metabase: Cache primary bucket ... ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ expired │ primary │ │ sec/op │ sec/op vs base │ Select/string_equal-8 3.529m ± 11% 3.689m ± 7% +4.55% (p=0.023 n=10) Select/string_not_equal-8 3.440m ± 7% 3.543m ± 13% ~ (p=0.190 n=10) Select/common_prefix-8 3.240m ± 6% 3.050m ± 5% -5.85% (p=0.005 n=10) Select/unknown-8 3.198m ± 6% 2.928m ± 8% -8.44% (p=0.003 n=10) geomean 3.349m 3.287m -1.84% │ expired │ primary │ │ B/op │ B/op vs base │ Select/string_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/string_not_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/common_prefix-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/unknown-8 1.877Mi ± 0% 1.779Mi ± 0% -5.26% (p=0.000 n=10) geomean 1.883Mi 1.784Mi -5.24% │ expired │ primary │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/string_not_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/common_prefix-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/unknown-8 45.05k ± 0% 42.05k ± 0% -6.65% (p=0.000 n=10) geomean 45.79k 42.79k -6.54% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:01 +00:00
Evgenii Stratonikov	21bed3362c	[#1685] metabase: Cache expired bucket ... ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ master │ expired │ │ sec/op │ sec/op vs base │ Select/string_equal-8 4.007m ± 10% 3.529m ± 11% -11.94% (p=0.000 n=10) Select/string_not_equal-8 3.834m ± 12% 3.440m ± 7% -10.29% (p=0.029 n=10) Select/common_prefix-8 3.470m ± 9% 3.240m ± 6% ~ (p=0.105 n=10) Select/unknown-8 3.156m ± 3% 3.198m ± 6% ~ (p=0.631 n=10) geomean 3.602m 3.349m -7.03% │ master │ expired │ │ B/op │ B/op vs base │ Select/string_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/string_not_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/common_prefix-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/unknown-8 1.900Mi ± 0% 1.877Mi ± 0% -1.18% (p=0.000 n=10) geomean 1.905Mi 1.883Mi -1.18% │ master │ expired │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/string_not_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/common_prefix-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/unknown-8 46.04k ± 0% 45.05k ± 0% -2.16% (p=0.000 n=10) geomean 46.78k 45.79k -2.13% ``` Change-Id: I9c7a5e1f5c8b9eb3f25a563fd74c6ad2a9d1b92e Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:01 +00:00
Evgenii Stratonikov	a49f0717b3	[#1685] metabase: Cache frequently accessed singleton buckets ... There are some buckets we access almost always, to check whether an object is alive. In search we also iterate over lots of objects, and `tx.Bucket()` shows itself a lot in pprof. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ 1 │ 2 │ │ sec/op │ sec/op vs base │ Select/string_equal-8 4.753m ± 6% 3.969m ± 14% -16.50% (p=0.000 n=10) Select/string_not_equal-8 4.247m ± 9% 3.486m ± 11% -17.93% (p=0.000 n=10) Select/common_prefix-8 4.163m ± 5% 3.323m ± 5% -20.18% (p=0.000 n=10) Select/unknown-8 3.557m ± 3% 3.064m ± 8% -13.85% (p=0.001 n=10) geomean 4.158m 3.445m -17.15% │ 1 │ 2 │ │ B/op │ B/op vs base │ Select/string_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/string_not_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/common_prefix-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/unknown-8 2.243Mi ± 0% 1.900Mi ± 0% -15.29% (p=0.000 n=10) geomean 2.248Mi 1.905Mi -15.26% │ 1 │ 2 │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/string_not_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/common_prefix-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/unknown-8 55.03k ± 0% 46.04k ± 0% -16.34% (p=0.000 n=10) geomean 55.78k 46.78k -16.12% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 10:17:42 +00:00
Airat Arifullin	a7ac30da9c	[#1642] tree: Refactor getSortedSubTree ... * Reuse `item` as result for `forest.TreeSortedByFilename` invocation. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	39f549a7ab	[#1642] tree: Intoduce a helper LastChild ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	760b6a44ea	[#1642] tree: Fix sorted getSubtree for multiversion filenames ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	a11b2d27e4	[#1642] tree: Introduce Cursor type ... * Use `Cursor` as parameter for `TreeSortedByFilename` Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Evgenii Stratonikov	a405fb1f39	[#1683] metabase: Check object status once in Select() ... objectStatus() is called twice for the same object: First, in selectObject() to filter removed objects. Then, again, in getObjectForSlowFilters() via db.get(). The second call will return the same result, so remove useless branch. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ status │ │ sec/op │ sec/op vs base │ Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10) Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10) Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10) Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10) geomean 5.045m 3.865m -23.39% │ old │ status │ │ B/op │ B/op vs base │ Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10) geomean 2.683Mi 2.248Mi -16.21% │ old │ status │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10) geomean 68.78k 55.77k -18.90% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-18 11:48:51 +00:00
Evgenii Stratonikov	a7319bc979	[#1683] metabase/test: Report allocs in benchmarkSelect() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-18 11:48:51 +00:00
Dmitrii Stepanov	07a660fbc4	[#1677] writecache: Add QoS limiter usage ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-14 16:23:33 +03:00
Dmitrii Stepanov	7893d763d1	[#1673] logger: Add sampling for journald logger ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-13 14:52:03 +03:00
Dmitrii Stepanov	ff4e9b6ae1	[#1673] logger: Drop unused fields ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-13 14:52:02 +03:00
Alexander Chuprov	997759994a	[#1676] golangci: Enable gci linter ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2025-03-13 12:04:01 +03:00
Evgenii Stratonikov	ecb6b0793c	[#1671] Use slices.ContainsFunc() where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	460e5cbccf	[#1671] Use slices.Delete() where possible ... gopatch is missing for this one, because https://github.com/uber-go/gopatch/issues/179 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	155d3ddb6e	[#1671] Use min builtin where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	40536d8a06	[#1671] Use fmt.Appendf where warranted ... Fix gopls warnings: ``` cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf ```` gopatch: ``` @@ var f expression @@ -[]byte(fmt.Sprintf(f, ...)) +fmt.Appendf(nil, f, ...) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	bcc84c85a0	[#1671] Replace interface{} with any ... gopatch: ``` @@ @@ -interface{} +any ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Dmitrii Stepanov	2005fdda09	[#1667] shard: Drop shard pool ... After adding an ops limiter, shard's `put` pool is redundant. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 13:59:51 +03:00
Dmitrii Stepanov	597bce7a87	[#1653] treeSvc: Add operations by IO tag metric ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	4ed2bbdb0f	[#1653] objectSvc: Add operations by IO tag metric ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	3727d60331	[#1653] qos: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	8643e0abc5	[#1668] writecache: Use object size to check free space ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-10 17:52:57 +03:00
Ekaterina Lebedeva	bd61f7bf0a	[#1666] audit: Fix duplicated log in Patch method ... When we do `object patch` with audit enabled we get several duplicated entries in logs. `object patch` request is logged in 2 places: 1. `(*auditPatchStream) CloseAndRecv()` - when the client closes the request stream or when stream gets aborted. 2. `(*auditPatchStream) Send()` - when stream was NOT aborted. `Send()` doesn't check if `err != nil` before logging. It led to to logging on every `Send()` call. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 13:27:07 +00:00
Ekaterina Lebedeva	9e31cb249f	[#1635] control: Add method to search shards by object ... Added method `ListShardsForObject` to ControlService and to StorageEngine. It returns information about shards storing object on the node. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 14:32:01 +03:00
Evgenii Stratonikov	6260d703ce	[#1665] treesvc: Disable service config query ... By default, gRPC fetches TXT report while resolving a domain. 0914bba6c5/internal/resolver/dns/dns_resolver.go (L336) This leads to a hanging dial if DNS is unavailable, even though the host may be specified in `/etc/hosts` (hello, localhost!). SDK client for the main API uses these options by default. Refs TrueCloudLab/frostfs-sdk-go#342 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-06 15:15:31 +03:00
Ekaterina Lebedeva	471aeeaff3	[#1659] audit: Fix duplicated request logs ... When we do `object put` with audit enabled we get several entries in logs: with and without object id. `object put` request is logged in 2 places: 1. `(*auditPutStream) CloseAndRecv()` - when the client closes the request stream or when stream gets aborted. 2. `(*auditPutStream) Send()` - when stream was NOT aborted. `Send()` does error check for `ErrAbortStream` because if there is any other error - CloseAndRecv will not be called and there won't be any audit log about failed request. It led to logging on every object chunck put, even if `err == nil`. Added check for `err != nil` in `Send()` to fix it. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-06 12:36:16 +03:00
Airat Arifullin	4c8f9580a1	[#1662] object: Fix CloseAndRecv for patch streamer ... * A client may open stream to server, not send anything and close the open stream immediatly. This shouldn't cause a panic; * Return the error if `s.patcher` is uninitialized. Uninitialized patcher cannot be closed, this causes a panic. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-05 17:16:10 +03:00
Dmitrii Stepanov	4685afb1dc	[#1636] engine: Validate limiter release in unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:30 +03:00
Dmitrii Stepanov	eb8b9b2b3b	[#1636] blobovniczatree: Validate limiter release in rebuild unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:30 +03:00
Dmitrii Stepanov	6c6e463b73	[#1636] shard: Change ops limiter on shard reload ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:29 +03:00
Dmitrii Stepanov	c2d855aedd	[#1636] qos: Return Resource Exhausted error ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:28 +03:00
Dmitrii Stepanov	2162f8e189	[#1636] object: Fix IO tag adjustment for Put/Patch ... There was no tag adjustment for CloseAndRecv. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:28 +03:00
Dmitrii Stepanov	b9360be1dc	[#1636] blobovniczatree: Use RebuildLimiter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:27 +03:00
Dmitrii Stepanov	ceff5e1f6a	[#1636] storage: Refactor shard rebuild ... Drop redundant interfaces. Rename fields. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:27 +03:00
Dmitrii Stepanov	e0dc3c3d0c	[#1636] shard: Add limiter usage ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:26 +03:00
Dmitrii Stepanov	0991077cb3	[#1657] engine: Fix data race in evacuation tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 14:14:12 +00:00
Aleksey Savchuk	dce269c62e	[#1639] services/object: Remove limiting pools for Put operation ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2025-02-28 11:08:09 +00:00
Airat Arifullin	9a0507704a	[#1651] apemanager: Wrap some errors with InvalidArgument status ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-02-27 18:25:09 +00:00
Evgenii Stratonikov	003d568ae2	[#1628] innerring: Relax container homomorphic hashing check ... Our initial desire was to prohibit using homomorphic hashing on the network level because of the resource consumption. However, the ability to use it, doesn't mean that we must. So only fail validation if container wants to have homomorphic hashing, but the network prohibits it. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-02-27 10:20:08 +03:00
Aleksey Savchuk	02f3a7f65c	[#1648] writecache: Fix race condition when reporting cache size metrics ... There is a race condition when multiple cache operation try to report the cache size metrics simultaneously. Consider the following example: - the initial total size of objects stored in the cache size is 2 - worker X deletes an object and reads the cache size, which is 1 - worker Y deletes an object and reads the cache size, which is 0 - worker Y reports the cache size it learnt, which is 0 - worker X reports the cache size it learnt, which is 1 As a result, the observed cache size is 1 (i. e. one object remains in the cache), which is incorrect because the actual cache size is 0. To fix this, let's report the metrics periodically in the flush loop. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2025-02-19 17:05:40 +03:00
Dmitrii Stepanov	5d79abe523	[#1608] shard: Add IO tag for rebuild ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-07 15:43:07 +03:00
Dmitrii Stepanov	0671c277db	[#1608] tree: Add IO tag for tree sync requests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-07 15:43:06 +03:00
Dmitrii Stepanov	92450a76ba	[#1608] shard: Add IO tag to GC ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-07 15:43:06 +03:00