5388 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Evgenii Stratonikov	1ec5377241	m ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 16:15:13 +03:00
Evgenii Stratonikov	eb8b0a6999	WIP: metabase: Find attribute without full unmarshal ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 16:15:12 +03:00
Evgenii Stratonikov	e8801dbf49	[#1691] metabase: Move cheaper conditions to the front in ListWithCursor() ... `objectLocked` call is expensive, it does IO. We may omit it if object is not expired. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:36 +00:00
Evgenii Stratonikov	eb9df85b98	[#1685] metabase: Cache primary bucket ... ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ expired │ primary │ │ sec/op │ sec/op vs base │ Select/string_equal-8 3.529m ± 11% 3.689m ± 7% +4.55% (p=0.023 n=10) Select/string_not_equal-8 3.440m ± 7% 3.543m ± 13% ~ (p=0.190 n=10) Select/common_prefix-8 3.240m ± 6% 3.050m ± 5% -5.85% (p=0.005 n=10) Select/unknown-8 3.198m ± 6% 2.928m ± 8% -8.44% (p=0.003 n=10) geomean 3.349m 3.287m -1.84% │ expired │ primary │ │ B/op │ B/op vs base │ Select/string_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/string_not_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/common_prefix-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/unknown-8 1.877Mi ± 0% 1.779Mi ± 0% -5.26% (p=0.000 n=10) geomean 1.883Mi 1.784Mi -5.24% │ expired │ primary │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/string_not_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/common_prefix-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/unknown-8 45.05k ± 0% 42.05k ± 0% -6.65% (p=0.000 n=10) geomean 45.79k 42.79k -6.54% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:01 +00:00
Evgenii Stratonikov	21bed3362c	[#1685] metabase: Cache expired bucket ... ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ master │ expired │ │ sec/op │ sec/op vs base │ Select/string_equal-8 4.007m ± 10% 3.529m ± 11% -11.94% (p=0.000 n=10) Select/string_not_equal-8 3.834m ± 12% 3.440m ± 7% -10.29% (p=0.029 n=10) Select/common_prefix-8 3.470m ± 9% 3.240m ± 6% ~ (p=0.105 n=10) Select/unknown-8 3.156m ± 3% 3.198m ± 6% ~ (p=0.631 n=10) geomean 3.602m 3.349m -7.03% │ master │ expired │ │ B/op │ B/op vs base │ Select/string_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/string_not_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/common_prefix-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/unknown-8 1.900Mi ± 0% 1.877Mi ± 0% -1.18% (p=0.000 n=10) geomean 1.905Mi 1.883Mi -1.18% │ master │ expired │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/string_not_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/common_prefix-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/unknown-8 46.04k ± 0% 45.05k ± 0% -2.16% (p=0.000 n=10) geomean 46.78k 45.79k -2.13% ``` Change-Id: I9c7a5e1f5c8b9eb3f25a563fd74c6ad2a9d1b92e Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:01 +00:00
Dmitrii Stepanov	af5b3575d0	[#1690] qos: Do not export zero metrics counters ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-20 14:42:35 +03:00
Evgenii Stratonikov	a49f0717b3	[#1685] metabase: Cache frequently accessed singleton buckets ... There are some buckets we access almost always, to check whether an object is alive. In search we also iterate over lots of objects, and `tx.Bucket()` shows itself a lot in pprof. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ 1 │ 2 │ │ sec/op │ sec/op vs base │ Select/string_equal-8 4.753m ± 6% 3.969m ± 14% -16.50% (p=0.000 n=10) Select/string_not_equal-8 4.247m ± 9% 3.486m ± 11% -17.93% (p=0.000 n=10) Select/common_prefix-8 4.163m ± 5% 3.323m ± 5% -20.18% (p=0.000 n=10) Select/unknown-8 3.557m ± 3% 3.064m ± 8% -13.85% (p=0.001 n=10) geomean 4.158m 3.445m -17.15% │ 1 │ 2 │ │ B/op │ B/op vs base │ Select/string_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/string_not_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/common_prefix-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/unknown-8 2.243Mi ± 0% 1.900Mi ± 0% -15.29% (p=0.000 n=10) geomean 2.248Mi 1.905Mi -15.26% │ 1 │ 2 │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/string_not_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/common_prefix-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/unknown-8 55.03k ± 0% 46.04k ± 0% -16.34% (p=0.000 n=10) geomean 55.78k 46.78k -16.12% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 10:17:42 +00:00
Airat Arifullin	a7ac30da9c	[#1642] tree: Refactor getSortedSubTree ... * Reuse `item` as result for `forest.TreeSortedByFilename` invocation. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	39f549a7ab	[#1642] tree: Intoduce a helper LastChild ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	760b6a44ea	[#1642] tree: Fix sorted getSubtree for multiversion filenames ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	a11b2d27e4	[#1642] tree: Introduce Cursor type ... * Use `Cursor` as parameter for `TreeSortedByFilename` Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Evgenii Stratonikov	a405fb1f39	[#1683] metabase: Check object status once in Select() ... objectStatus() is called twice for the same object: First, in selectObject() to filter removed objects. Then, again, in getObjectForSlowFilters() via db.get(). The second call will return the same result, so remove useless branch. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ status │ │ sec/op │ sec/op vs base │ Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10) Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10) Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10) Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10) geomean 5.045m 3.865m -23.39% │ old │ status │ │ B/op │ B/op vs base │ Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10) geomean 2.683Mi 2.248Mi -16.21% │ old │ status │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10) geomean 68.78k 55.77k -18.90% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-18 11:48:51 +00:00
Evgenii Stratonikov	a7319bc979	[#1683] metabase/test: Report allocs in benchmarkSelect() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-18 11:48:51 +00:00
Alexander Chuprov	fc743cc537	[#1684] cli: Correct description of control shards writecache seal ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2025-03-18 10:56:30 +03:00
Evgenii Stratonikov	54ef71a92f	[#1680] Update staticcheck to 2025.1.1 ... Change-Id: Ie851e714afebf171c4d42d4c49b42379c2665113 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-17 10:02:53 +00:00
Evgenii Stratonikov	91c7b39232	[#1680] go.mod: Bump go version to 1.23 ... Change-Id: I77f908924f675e676f0db6a57204d7c1e0df219a Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-17 10:02:53 +00:00
Evgenii Stratonikov	ef6ac751df	[#1671] Update gopls to v0.17.1 ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-17 10:02:53 +00:00
Dmitrii Stepanov	fde2649e60	[#1678] adm: Fix frostfs-adm morph list-subjects & list-group-subjects ... `include-names` for `list-subjects` returns error `invalid response subject struct` because `ListSubjects` returns only subject addresses (see frostfs-contract). Replace `include-names` with `extended` as now all subject info printed. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-14 13:42:29 +00:00
Dmitrii Stepanov	07a660fbc4	[#1677] writecache: Add QoS limiter usage ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-14 16:23:33 +03:00
Dmitrii Stepanov	7893d763d1	[#1673] logger: Add sampling for journald logger ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-13 14:52:03 +03:00
Dmitrii Stepanov	ff4e9b6ae1	[#1673] logger: Drop unused fields ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-13 14:52:02 +03:00
Alexander Chuprov	997759994a	[#1676] golangci: Enable gci linter ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2025-03-13 12:04:01 +03:00
Evgenii Stratonikov	ecb6b0793c	[#1671] Use slices.ContainsFunc() where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	460e5cbccf	[#1671] Use slices.Delete() where possible ... gopatch is missing for this one, because https://github.com/uber-go/gopatch/issues/179 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	155d3ddb6e	[#1671] Use min builtin where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	40536d8a06	[#1671] Use fmt.Appendf where warranted ... Fix gopls warnings: ``` cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf ```` gopatch: ``` @@ var f expression @@ -[]byte(fmt.Sprintf(f, ...)) +fmt.Appendf(nil, f, ...) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	d66bffb191	[#1671] Use max builtin where possible ... gopatcH: ``` @@ var d, a expression @@ -if d < a { - d = a -} -return d +return max(d, a) @@ var d, a expression @@ -if d <= a { - d = a -} -return d +return max(d, a) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	bcc84c85a0	[#1671] Replace interface{} with any ... gopatch: ``` @@ @@ -interface{} +any ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Dmitrii Stepanov	737788b35f	[#1669] go.mod: Bump frostfs-qos version ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-12 09:23:53 +03:00
Dmitrii Stepanov	2005fdda09	[#1667] shard: Drop shard pool ... After adding an ops limiter, shard's `put` pool is redundant. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 13:59:51 +03:00
Dmitrii Stepanov	597bce7a87	[#1653] treeSvc: Add operations by IO tag metric ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	4ed2bbdb0f	[#1653] objectSvc: Add operations by IO tag metric ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	3727d60331	[#1653] qos: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	d36afa31c7	[#1653] qos: Fix logging ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	8643e0abc5	[#1668] writecache: Use object size to check free space ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-10 17:52:57 +03:00
Ekaterina Lebedeva	bd61f7bf0a	[#1666] audit: Fix duplicated log in Patch method ... When we do `object patch` with audit enabled we get several duplicated entries in logs. `object patch` request is logged in 2 places: 1. `(*auditPatchStream) CloseAndRecv()` - when the client closes the request stream or when stream gets aborted. 2. `(*auditPatchStream) Send()` - when stream was NOT aborted. `Send()` doesn't check if `err != nil` before logging. It led to to logging on every `Send()` call. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 13:27:07 +00:00
Ekaterina Lebedeva	df6d9da82a	[#1635] cli: Add command to get object's shard info ... Added `frostfs-cli object locate` subcommand. It lists info about shards storing an object. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 14:32:01 +03:00
Ekaterina Lebedeva	aab8addae0	[#1635] cli: Make object.readObjectAddress() public ... This method will be useful for upcoming control command. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 14:32:01 +03:00
Ekaterina Lebedeva	9e31cb249f	[#1635] control: Add method to search shards by object ... Added method `ListShardsForObject` to ControlService and to StorageEngine. It returns information about shards storing object on the node. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 14:32:01 +03:00
Evgenii Stratonikov	6260d703ce	[#1665] treesvc: Disable service config query ... By default, gRPC fetches TXT report while resolving a domain. 0914bba6c5/internal/resolver/dns/dns_resolver.go (L336) This leads to a hanging dial if DNS is unavailable, even though the host may be specified in `/etc/hosts` (hello, localhost!). SDK client for the main API uses these options by default. Refs TrueCloudLab/frostfs-sdk-go#342 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-06 15:15:31 +03:00
Evgenii Stratonikov	a17c3356fa	[#1665] go.mod: Update sdk-go ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-06 15:15:31 +03:00
Ekaterina Lebedeva	471aeeaff3	[#1659] audit: Fix duplicated request logs ... When we do `object put` with audit enabled we get several entries in logs: with and without object id. `object put` request is logged in 2 places: 1. `(*auditPutStream) CloseAndRecv()` - when the client closes the request stream or when stream gets aborted. 2. `(*auditPutStream) Send()` - when stream was NOT aborted. `Send()` does error check for `ErrAbortStream` because if there is any other error - CloseAndRecv will not be called and there won't be any audit log about failed request. It led to logging on every object chunck put, even if `err == nil`. Added check for `err != nil` in `Send()` to fix it. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-06 12:36:16 +03:00
Airat Arifullin	4c8f9580a1	[#1662] object: Fix CloseAndRecv for patch streamer ... * A client may open stream to server, not send anything and close the open stream immediatly. This shouldn't cause a panic; * Return the error if `s.patcher` is uninitialized. Uninitialized patcher cannot be closed, this causes a panic. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-05 17:16:10 +03:00
Aleksey Savchuk	bf8914fedc	[#1416] lens/explorer: Support metabase schema v3 ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2025-03-05 07:20:05 +00:00
Evgenii Stratonikov	5ba0e2918e	[#1661] cli: Clarify --rpc-endpoint values ... `multiaddr` is not something an average user knows. Personally, I have never used it in CLI. On the other hand, we need to connect with TLS quite often, so it needs to be mentioned in help. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-04 10:19:49 +03:00
Dmitrii Stepanov	4685afb1dc	[#1636] engine: Validate limiter release in unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:30 +03:00
Dmitrii Stepanov	eb8b9b2b3b	[#1636] blobovniczatree: Validate limiter release in rebuild unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:30 +03:00
Dmitrii Stepanov	6c6e463b73	[#1636] shard: Change ops limiter on shard reload ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:29 +03:00
Dmitrii Stepanov	401d96a89e	[#1636] config: Refactor newConfig and oldConfig ... `newConfig` is actually target config to set config values from source (which is called `oldConfig`). Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:29 +03:00
Dmitrii Stepanov	8ed71a969e	[#1636] qos: Add semaphore limiter ... If no tags specified, then limiter could be optimized to use atomic semaphore. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-02-28 17:25:29 +03:00