5401 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Evgenii Stratonikov	e7e91ef634	[#1689] adm: Remove storagecfg subcommand ... It is unused and unsupported for a long time. Change-Id: I570567db4e8cb202e41286064406ad85cd0e7a39 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-21 15:05:02 +00:00
Aleksey Savchuk	4919b6a206	[#1689] node/config: Allow zero max_ops in RPC limits config ... The limiter allows zeros for limits, meaning "this operation is disabled". However, the config didn't allow zero due to the lack of distinction between "no value" and "zero" - cast functions read both `nil` and zero as zero. Now, the config allows a zero limit. Added tests. Managing such cases should be easier after #1610. Change-Id: Ifc840732390b2feb975f230573b34bf479406e05 Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2025-03-21 17:00:40 +03:00
Alexander Chuprov	d951289131	[#1294] docs: Fix description of shard switching mode ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2025-03-21 12:13:03 +00:00
Evgenii Stratonikov	016f2e11e3	[#1689] Makefile: Add more restricted .SHELLFLAGS ... Catch more errors immediately. Change-Id: I576f1b394a2b167c78c693a794ab8cca3ac1013b Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-21 11:49:49 +00:00
Evgenii Stratonikov	9aa486c9d8	[#1689] Makefile: Create dirs with -p flag ... On CI there is no `bin` directory initially, so an error occurs ``` mkdir: cannot create directory '/var/cache/jenkins-agent/workspace/gerrit/frostfs-node#55-488b12-8ac3c/bin/gofumpt': No such file or directory ``` Change-Id: I43895c8f5ed7cc5c71c8025228710279f9e75e9c Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-21 11:49:42 +00:00
Dmitrii Stepanov	af76350bfb	[#1695] qos: Sort tags by asc ... Change-Id: Ia23e392bb49d2536096de2ba07fc6f8fb7ac0489 Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-21 11:25:33 +00:00
Evgenii Stratonikov	3fa5c22ddf	[#1689] Makefile: Add default reviewers via --push-option ... Gerrit doesn't provide an easy way to have default reviewers assigned to new change requests. However, we can use `--push-option` and mention all people from storage-core-developers group. Change-Id: Ia01f8a3c5c8eb8a1dca6efb66fbe07018f6a42c9 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-21 11:25:21 +00:00
Dmitrii Stepanov	5385f9994f	[#1695] mod: Bump frostfs-observability version ... Change-Id: Id362b71f743ff70c8cd374030c9fa67e2566022f Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-21 13:28:02 +03:00
Dmitrii Stepanov	eea46a599d	[#1695] qos: Add treesync tag ... Tree sync is too much different from GC and rebuild to use the same tag for GC and tree sync. Change-Id: Ib44d5fa9a88daff507d759d0b0410cc9272e236f Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-21 13:28:02 +03:00
Anton Nikiforov	049a650b89	[#1619] logger: Simplify logger config reloading ... Change-Id: Ide892b250304b8cdb6c279f5f728c3b35f05df54 Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2025-03-21 09:03:54 +00:00
Evgenii Stratonikov	3f4717a37f	[#1692] metabase: Do not allocate map in cache unless needed ... Change-Id: I8b1015a8c7c3df4153a08fdb788117d9f0d6c333 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-21 08:56:32 +00:00
Evgenii Stratonikov	60cea8c714	[#1692] metabase/test: Fix end of iteration error check ... This is not good: ``` BenchmarkListWithCursor/1_item-8 --- FAIL: BenchmarkListWithCursor/1_item-8 list_test.go:63: error: end of object listing ``` Change-Id: I61b70937ce30fefaf16ebeb0cdb51bdd39096061 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-21 08:56:18 +00:00
Evgenii Stratonikov	7df2912a83	[#1689] Makefile: Create prepare-commit-msg hook too ... `commit-msg` is ignored when `--no-verify` option is used, so there is no way to ignore `pre-commit` while retaining `commit-msg` hook. Ignoring pre-commit is useful, though, so we might add Change-Id in `prepare-commit-msg` hook instead. It accepts more parameters, but the first one is a file with the commit message, so we may reuse `commit-msg` hook. Change-Id: I4edb79810bbe38a5dcf7f4f07535f34c6bda0da3 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 18:01:53 +03:00
Evgenii Stratonikov	affab25512	Makefile: Add Gerrit-related targets ... This commit adds helper targets to easily setup an existing repo for work with Gerrit. Change-Id: I0696eb8ea84cc16a9482be6a2fb0382fe624bb96 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 16:48:04 +03:00
Vitaliy Potyarkin	45b7796151	[#1689] ci: Reimplement CI tasks in Jenkinsfile ... This commit introduces Jenkins pipeline that duplicates the features of existing Forgejo Actions workflows. Change-Id: I657a6c27373a1ed4736ae27b4fb660e0ac86012d Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>	2025-03-20 12:55:30 +00:00
Evgenii Stratonikov	e8801dbf49	[#1691] metabase: Move cheaper conditions to the front in ListWithCursor() ... `objectLocked` call is expensive, it does IO. We may omit it if object is not expired. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:36 +00:00
Evgenii Stratonikov	eb9df85b98	[#1685] metabase: Cache primary bucket ... ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ expired │ primary │ │ sec/op │ sec/op vs base │ Select/string_equal-8 3.529m ± 11% 3.689m ± 7% +4.55% (p=0.023 n=10) Select/string_not_equal-8 3.440m ± 7% 3.543m ± 13% ~ (p=0.190 n=10) Select/common_prefix-8 3.240m ± 6% 3.050m ± 5% -5.85% (p=0.005 n=10) Select/unknown-8 3.198m ± 6% 2.928m ± 8% -8.44% (p=0.003 n=10) geomean 3.349m 3.287m -1.84% │ expired │ primary │ │ B/op │ B/op vs base │ Select/string_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/string_not_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/common_prefix-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10) Select/unknown-8 1.877Mi ± 0% 1.779Mi ± 0% -5.26% (p=0.000 n=10) geomean 1.883Mi 1.784Mi -5.24% │ expired │ primary │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/string_not_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/common_prefix-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10) Select/unknown-8 45.05k ± 0% 42.05k ± 0% -6.65% (p=0.000 n=10) geomean 45.79k 42.79k -6.54% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:01 +00:00
Evgenii Stratonikov	21bed3362c	[#1685] metabase: Cache expired bucket ... ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ master │ expired │ │ sec/op │ sec/op vs base │ Select/string_equal-8 4.007m ± 10% 3.529m ± 11% -11.94% (p=0.000 n=10) Select/string_not_equal-8 3.834m ± 12% 3.440m ± 7% -10.29% (p=0.029 n=10) Select/common_prefix-8 3.470m ± 9% 3.240m ± 6% ~ (p=0.105 n=10) Select/unknown-8 3.156m ± 3% 3.198m ± 6% ~ (p=0.631 n=10) geomean 3.602m 3.349m -7.03% │ master │ expired │ │ B/op │ B/op vs base │ Select/string_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/string_not_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/common_prefix-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10) Select/unknown-8 1.900Mi ± 0% 1.877Mi ± 0% -1.18% (p=0.000 n=10) geomean 1.905Mi 1.883Mi -1.18% │ master │ expired │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/string_not_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/common_prefix-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10) Select/unknown-8 46.04k ± 0% 45.05k ± 0% -2.16% (p=0.000 n=10) geomean 46.78k 45.79k -2.13% ``` Change-Id: I9c7a5e1f5c8b9eb3f25a563fd74c6ad2a9d1b92e Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 12:52:01 +00:00
Dmitrii Stepanov	af5b3575d0	[#1690] qos: Do not export zero metrics counters ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-20 14:42:35 +03:00
Evgenii Stratonikov	a49f0717b3	[#1685] metabase: Cache frequently accessed singleton buckets ... There are some buckets we access almost always, to check whether an object is alive. In search we also iterate over lots of objects, and `tx.Bucket()` shows itself a lot in pprof. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ 1 │ 2 │ │ sec/op │ sec/op vs base │ Select/string_equal-8 4.753m ± 6% 3.969m ± 14% -16.50% (p=0.000 n=10) Select/string_not_equal-8 4.247m ± 9% 3.486m ± 11% -17.93% (p=0.000 n=10) Select/common_prefix-8 4.163m ± 5% 3.323m ± 5% -20.18% (p=0.000 n=10) Select/unknown-8 3.557m ± 3% 3.064m ± 8% -13.85% (p=0.001 n=10) geomean 4.158m 3.445m -17.15% │ 1 │ 2 │ │ B/op │ B/op vs base │ Select/string_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/string_not_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/common_prefix-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10) Select/unknown-8 2.243Mi ± 0% 1.900Mi ± 0% -15.29% (p=0.000 n=10) geomean 2.248Mi 1.905Mi -15.26% │ 1 │ 2 │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/string_not_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/common_prefix-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10) Select/unknown-8 55.03k ± 0% 46.04k ± 0% -16.34% (p=0.000 n=10) geomean 55.78k 46.78k -16.12% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-20 10:17:42 +00:00
Airat Arifullin	a7ac30da9c	[#1642] tree: Refactor getSortedSubTree ... * Reuse `item` as result for `forest.TreeSortedByFilename` invocation. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	39f549a7ab	[#1642] tree: Intoduce a helper LastChild ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	760b6a44ea	[#1642] tree: Fix sorted getSubtree for multiversion filenames ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Airat Arifullin	a11b2d27e4	[#1642] tree: Introduce Cursor type ... * Use `Cursor` as parameter for `TreeSortedByFilename` Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2025-03-20 10:12:49 +00:00
Evgenii Stratonikov	a405fb1f39	[#1683] metabase: Check object status once in Select() ... objectStatus() is called twice for the same object: First, in selectObject() to filter removed objects. Then, again, in getObjectForSlowFilters() via db.get(). The second call will return the same result, so remove useless branch. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ status │ │ sec/op │ sec/op vs base │ Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10) Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10) Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10) Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10) geomean 5.045m 3.865m -23.39% │ old │ status │ │ B/op │ B/op vs base │ Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10) Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10) geomean 2.683Mi 2.248Mi -16.21% │ old │ status │ │ allocs/op │ allocs/op vs base │ Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10) Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10) geomean 68.78k 55.77k -18.90% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-18 11:48:51 +00:00
Evgenii Stratonikov	a7319bc979	[#1683] metabase/test: Report allocs in benchmarkSelect() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-18 11:48:51 +00:00
Alexander Chuprov	fc743cc537	[#1684] cli: Correct description of control shards writecache seal ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2025-03-18 10:56:30 +03:00
Evgenii Stratonikov	54ef71a92f	[#1680] Update staticcheck to 2025.1.1 ... Change-Id: Ie851e714afebf171c4d42d4c49b42379c2665113 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-17 10:02:53 +00:00
Evgenii Stratonikov	91c7b39232	[#1680] go.mod: Bump go version to 1.23 ... Change-Id: I77f908924f675e676f0db6a57204d7c1e0df219a Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-17 10:02:53 +00:00
Evgenii Stratonikov	ef6ac751df	[#1671] Update gopls to v0.17.1 ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-17 10:02:53 +00:00
Dmitrii Stepanov	fde2649e60	[#1678] adm: Fix frostfs-adm morph list-subjects & list-group-subjects ... `include-names` for `list-subjects` returns error `invalid response subject struct` because `ListSubjects` returns only subject addresses (see frostfs-contract). Replace `include-names` with `extended` as now all subject info printed. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-14 13:42:29 +00:00
Dmitrii Stepanov	07a660fbc4	[#1677] writecache: Add QoS limiter usage ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-14 16:23:33 +03:00
Dmitrii Stepanov	7893d763d1	[#1673] logger: Add sampling for journald logger ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-13 14:52:03 +03:00
Dmitrii Stepanov	ff4e9b6ae1	[#1673] logger: Drop unused fields ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-13 14:52:02 +03:00
Alexander Chuprov	997759994a	[#1676] golangci: Enable gci linter ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2025-03-13 12:04:01 +03:00
Evgenii Stratonikov	ecb6b0793c	[#1671] Use slices.ContainsFunc() where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	460e5cbccf	[#1671] Use slices.Delete() where possible ... gopatch is missing for this one, because https://github.com/uber-go/gopatch/issues/179 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	155d3ddb6e	[#1671] Use min builtin where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	40536d8a06	[#1671] Use fmt.Appendf where warranted ... Fix gopls warnings: ``` cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf ```` gopatch: ``` @@ var f expression @@ -[]byte(fmt.Sprintf(f, ...)) +fmt.Appendf(nil, f, ...) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	d66bffb191	[#1671] Use max builtin where possible ... gopatcH: ``` @@ var d, a expression @@ -if d < a { - d = a -} -return d +return max(d, a) @@ var d, a expression @@ -if d <= a { - d = a -} -return d +return max(d, a) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Evgenii Stratonikov	bcc84c85a0	[#1671] Replace interface{} with any ... gopatch: ``` @@ @@ -interface{} +any ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2025-03-13 08:12:20 +00:00
Dmitrii Stepanov	737788b35f	[#1669] go.mod: Bump frostfs-qos version ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-12 09:23:53 +03:00
Dmitrii Stepanov	2005fdda09	[#1667] shard: Drop shard pool ... After adding an ops limiter, shard's `put` pool is redundant. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 13:59:51 +03:00
Dmitrii Stepanov	597bce7a87	[#1653] treeSvc: Add operations by IO tag metric ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	4ed2bbdb0f	[#1653] objectSvc: Add operations by IO tag metric ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	3727d60331	[#1653] qos: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	d36afa31c7	[#1653] qos: Fix logging ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-11 10:57:47 +00:00
Dmitrii Stepanov	8643e0abc5	[#1668] writecache: Use object size to check free space ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2025-03-10 17:52:57 +03:00
Ekaterina Lebedeva	bd61f7bf0a	[#1666] audit: Fix duplicated log in Patch method ... When we do `object patch` with audit enabled we get several duplicated entries in logs. `object patch` request is logged in 2 places: 1. `(*auditPatchStream) CloseAndRecv()` - when the client closes the request stream or when stream gets aborted. 2. `(*auditPatchStream) Send()` - when stream was NOT aborted. `Send()` doesn't check if `err != nil` before logging. It led to to logging on every `Send()` call. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 13:27:07 +00:00
Ekaterina Lebedeva	df6d9da82a	[#1635] cli: Add command to get object's shard info ... Added `frostfs-cli object locate` subcommand. It lists info about shards storing an object. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2025-03-07 14:32:01 +03:00