d42f67e053
[ #1656 ] qos: Add tests for AdjustOutgoingIOTag Interceptors
...
Change-Id: If534e756b26cf7f202039d48ecdf554b4283728b
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-26 09:24:25 +03:00
99340b2717
[ #1656 ] qos: Add test for SetCriticalIOTag Interceptor
...
Change-Id: I4a55fcb84e6f65408a1c0120ac917e49e23354a1
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-26 09:24:22 +03:00
c16788f9c6
[ #1656 ] qos: Add tests for MaxActiveRPCLimiter Interceptors
...
Change-Id: Ib65890ae5aec34c34e15d4ec1f05952f74f1ad26
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-26 09:24:19 +03:00
163e2e9f83
ci: Cache pre-commit installations on Jenkins Agent
...
This change introduces a custom helper from our shared library [0]
that defines ad-hoc container environment to execute CI steps in.
Pre-commit installation will now be cached on Jenkins Agent: builds will
tolerate network hiccups better and we will also save some run time
(although on non-critical path of a parallel process).
[0]: TrueCloudLab/jenkins#8
Change-Id: I93b01f169c457aa35f4d8bc5b90f31b31e2bd8b2
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-24 17:04:24 +03:00
0c664fa804
[ #1689 ] qos: Fix metric description
...
Change-Id: I460fdd3713e765d57ef3ff2945b9b3776f46c164
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-24 09:42:45 +00:00
0a9d139e20
[ #1689 ] morph/client: Remove notary hash field from notaryInfo
...
Notary contract hash is constant.
Change-Id: I7935580acbced5c9d567875ea75daa57cc259a3c
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-24 09:20:19 +00:00
3bb1fb744a
[ #1689 ] morph/client: Reuse auto-generated wrappers for NNS
...
Make code simpler, remove unused methods.
Change-Id: I18807f2c14b5a96e533e5e3fc153e23c742c66c1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-24 09:20:19 +00:00
ccdd6cb767
[ #1052 ] object: Nuke out acl middleware
...
* Remove `acl` package as it's no longer used;
* Remove `RequestContext`;
* Fix `cmd/frostfs-node`.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-23 06:39:32 +00:00
73e35bc885
[ #1052 ] object: Make ape middleware form request info
...
* Move some helpers from `acl/v2` package to `ape`. Also move errors;
* Introduce `Metadata`, `RequestInfo` types;
* Introduce `RequestInfoExtractor` interface and its implementation.
The extractor's purpose is to extract request info based on request
metadata. It also validates session token;
* Refactor ape service - each handler forms request info and pass
necessary fields to checker.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-23 06:39:32 +00:00
eed0824590
go.mod: Bump frostfs-qos version
...
Change-Id: I8bc045b509ee1259cfad288477a0b7d045683f10
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 16:36:03 +00:00
a4da1da767
[ #905 ] morph/client: Fetch NNS hash once on init
...
NNS contract hash is taken from the contract with ID=1.
Because morph client is expected to work with the same chain,
and because contract hash doesn't change on update, there is no need to
fetch it from each new endpoint.
Change-Id: Ic6dc18283789da076d6a0b3701139b97037714cc
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:13:54 +00:00
30099194ba
[ #1689 ] config: Remove testnet and mainnet configs
...
They are invalid and unsupported.
There is neither mainnet nor testnet currently.
Change-Id: I520363e2de0c22a584238accc253248be3eefea5
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:07:02 +00:00
e7e91ef634
[ #1689 ] adm: Remove storagecfg subcommand
...
It is unused and unsupported for a long time.
Change-Id: I570567db4e8cb202e41286064406ad85cd0e7a39
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:05:02 +00:00
4919b6a206
[ #1689 ] node/config: Allow zero max_ops in RPC limits config
...
The limiter allows zeros for limits, meaning "this operation is
disabled". However, the config didn't allow zero due to the lack of
distinction between "no value" and "zero" - cast functions read both
`nil` and zero as zero.
Now, the config allows a zero limit. Added tests.
Managing such cases should be easier after #1610 .
Change-Id: Ifc840732390b2feb975f230573b34bf479406e05
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-03-21 17:00:40 +03:00
d951289131
[ #1294 ] docs: Fix description of shard switching mode
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-21 12:13:03 +00:00
016f2e11e3
[ #1689 ] Makefile: Add more restricted .SHELLFLAGS
...
Catch more errors immediately.
Change-Id: I576f1b394a2b167c78c693a794ab8cca3ac1013b
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:49:49 +00:00
9aa486c9d8
[ #1689 ] Makefile: Create dirs with -p flag
...
On CI there is no `bin` directory initially, so an error occurs
```
mkdir: cannot create directory '/var/cache/jenkins-agent/workspace/gerrit/frostfs-node#55-488b12-8ac3c/bin/gofumpt': No such file or directory
```
Change-Id: I43895c8f5ed7cc5c71c8025228710279f9e75e9c
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:49:42 +00:00
af76350bfb
[ #1695 ] qos: Sort tags by asc
...
Change-Id: Ia23e392bb49d2536096de2ba07fc6f8fb7ac0489
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 11:25:33 +00:00
3fa5c22ddf
[ #1689 ] Makefile: Add default reviewers via --push-option
...
Gerrit doesn't provide an easy way to have default reviewers assigned to
new change requests. However, we can use `--push-option` and mention all
people from storage-core-developers group.
Change-Id: Ia01f8a3c5c8eb8a1dca6efb66fbe07018f6a42c9
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:25:21 +00:00
5385f9994f
[ #1695 ] mod: Bump frostfs-observability version
...
Change-Id: Id362b71f743ff70c8cd374030c9fa67e2566022f
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:02 +03:00
eea46a599d
[ #1695 ] qos: Add treesync tag
...
Tree sync is too much different from GC and rebuild to use the same tag for GC and tree sync.
Change-Id: Ib44d5fa9a88daff507d759d0b0410cc9272e236f
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:02 +03:00
049a650b89
[ #1619 ] logger: Simplify logger config reloading
...
Change-Id: Ide892b250304b8cdb6c279f5f728c3b35f05df54
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-21 09:03:54 +00:00
3f4717a37f
[ #1692 ] metabase: Do not allocate map in cache unless needed
...
Change-Id: I8b1015a8c7c3df4153a08fdb788117d9f0d6c333
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 08:56:32 +00:00
60cea8c714
[ #1692 ] metabase/test: Fix end of iteration error check
...
This is not good:
```
BenchmarkListWithCursor/1_item-8 --- FAIL: BenchmarkListWithCursor/1_item-8
list_test.go:63: error: end of object listing
```
Change-Id: I61b70937ce30fefaf16ebeb0cdb51bdd39096061
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 08:56:18 +00:00
7df2912a83
[ #1689 ] Makefile: Create prepare-commit-msg hook too
...
`commit-msg` is ignored when `--no-verify` option is used, so there is
no way to ignore `pre-commit` while retaining `commit-msg` hook.
Ignoring pre-commit is useful, though, so we might add Change-Id in
`prepare-commit-msg` hook instead. It accepts more parameters, but the
first one is a file with the commit message, so we may reuse
`commit-msg` hook.
Change-Id: I4edb79810bbe38a5dcf7f4f07535f34c6bda0da3
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 18:01:53 +03:00
affab25512
Makefile: Add Gerrit-related targets
...
This commit adds helper targets to easily setup an existing repo for
work with Gerrit.
Change-Id: I0696eb8ea84cc16a9482be6a2fb0382fe624bb96
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 16:48:04 +03:00
45b7796151
[ #1689 ] ci: Reimplement CI tasks in Jenkinsfile
...
This commit introduces Jenkins pipeline that duplicates the features of
existing Forgejo Actions workflows.
Change-Id: I657a6c27373a1ed4736ae27b4fb660e0ac86012d
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-20 12:55:30 +00:00
e8801dbf49
[ #1691 ] metabase: Move cheaper conditions to the front in ListWithCursor()
...
`objectLocked` call is expensive, it does IO. We may omit it if object
is not expired.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:36 +00:00
eb9df85b98
[ #1685 ] metabase: Cache primary bucket
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ expired │ primary │
│ sec/op │ sec/op vs base │
Select/string_equal-8 3.529m ± 11% 3.689m ± 7% +4.55% (p=0.023 n=10)
Select/string_not_equal-8 3.440m ± 7% 3.543m ± 13% ~ (p=0.190 n=10)
Select/common_prefix-8 3.240m ± 6% 3.050m ± 5% -5.85% (p=0.005 n=10)
Select/unknown-8 3.198m ± 6% 2.928m ± 8% -8.44% (p=0.003 n=10)
geomean 3.349m 3.287m -1.84%
│ expired │ primary │
│ B/op │ B/op vs base │
Select/string_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/string_not_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/common_prefix-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/unknown-8 1.877Mi ± 0% 1.779Mi ± 0% -5.26% (p=0.000 n=10)
geomean 1.883Mi 1.784Mi -5.24%
│ expired │ primary │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/string_not_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/common_prefix-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/unknown-8 45.05k ± 0% 42.05k ± 0% -6.65% (p=0.000 n=10)
geomean 45.79k 42.79k -6.54%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:01 +00:00
21bed3362c
[ #1685 ] metabase: Cache expired bucket
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ master │ expired │
│ sec/op │ sec/op vs base │
Select/string_equal-8 4.007m ± 10% 3.529m ± 11% -11.94% (p=0.000 n=10)
Select/string_not_equal-8 3.834m ± 12% 3.440m ± 7% -10.29% (p=0.029 n=10)
Select/common_prefix-8 3.470m ± 9% 3.240m ± 6% ~ (p=0.105 n=10)
Select/unknown-8 3.156m ± 3% 3.198m ± 6% ~ (p=0.631 n=10)
geomean 3.602m 3.349m -7.03%
│ master │ expired │
│ B/op │ B/op vs base │
Select/string_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/string_not_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/common_prefix-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/unknown-8 1.900Mi ± 0% 1.877Mi ± 0% -1.18% (p=0.000 n=10)
geomean 1.905Mi 1.883Mi -1.18%
│ master │ expired │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/string_not_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/common_prefix-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/unknown-8 46.04k ± 0% 45.05k ± 0% -2.16% (p=0.000 n=10)
geomean 46.78k 45.79k -2.13%
```
Change-Id: I9c7a5e1f5c8b9eb3f25a563fd74c6ad2a9d1b92e
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:01 +00:00
af5b3575d0
[ #1690 ] qos: Do not export zero metrics counters
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-20 14:42:35 +03:00
a49f0717b3
[ #1685 ] metabase: Cache frequently accessed singleton buckets
...
There are some buckets we access almost always, to check whether an
object is alive. In search we also iterate over lots of objects, and
`tx.Bucket()` shows itself a lot in pprof.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ 1 │ 2 │
│ sec/op │ sec/op vs base │
Select/string_equal-8 4.753m ± 6% 3.969m ± 14% -16.50% (p=0.000 n=10)
Select/string_not_equal-8 4.247m ± 9% 3.486m ± 11% -17.93% (p=0.000 n=10)
Select/common_prefix-8 4.163m ± 5% 3.323m ± 5% -20.18% (p=0.000 n=10)
Select/unknown-8 3.557m ± 3% 3.064m ± 8% -13.85% (p=0.001 n=10)
geomean 4.158m 3.445m -17.15%
│ 1 │ 2 │
│ B/op │ B/op vs base │
Select/string_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/string_not_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/common_prefix-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/unknown-8 2.243Mi ± 0% 1.900Mi ± 0% -15.29% (p=0.000 n=10)
geomean 2.248Mi 1.905Mi -15.26%
│ 1 │ 2 │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/string_not_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/common_prefix-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/unknown-8 55.03k ± 0% 46.04k ± 0% -16.34% (p=0.000 n=10)
geomean 55.78k 46.78k -16.12%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 10:17:42 +00:00
a7ac30da9c
[ #1642 ] tree: Refactor getSortedSubTree
...
* Reuse `item` as result for `forest.TreeSortedByFilename`
invocation.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
39f549a7ab
[ #1642 ] tree: Intoduce a helper LastChild
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
760b6a44ea
[ #1642 ] tree: Fix sorted getSubtree for multiversion filenames
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a11b2d27e4
[ #1642 ] tree: Introduce Cursor type
...
* Use `Cursor` as parameter for `TreeSortedByFilename`
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a405fb1f39
[ #1683 ] metabase: Check object status once in Select()
...
objectStatus() is called twice for the same object:
First, in selectObject() to filter removed objects.
Then, again, in getObjectForSlowFilters() via db.get().
The second call will return the same result, so remove useless branch.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ status │
│ sec/op │ sec/op vs base │
Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10)
Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10)
Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10)
Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10)
geomean 5.045m 3.865m -23.39%
│ old │ status │
│ B/op │ B/op vs base │
Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10)
geomean 2.683Mi 2.248Mi -16.21%
│ old │ status │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10)
geomean 68.78k 55.77k -18.90%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
a7319bc979
[ #1683 ] metabase/test: Report allocs in benchmarkSelect()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
fc743cc537
[ #1684 ] cli: Correct description of control shards writecache seal
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-18 10:56:30 +03:00
54ef71a92f
[ #1680 ] Update staticcheck to 2025.1.1
...
Change-Id: Ie851e714afebf171c4d42d4c49b42379c2665113
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
91c7b39232
[ #1680 ] go.mod: Bump go version to 1.23
...
Change-Id: I77f908924f675e676f0db6a57204d7c1e0df219a
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
ef6ac751df
[ #1671 ] Update gopls to v0.17.1
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
fde2649e60
[ #1678 ] adm: Fix frostfs-adm morph list-subjects & list-group-subjects
...
`include-names` for `list-subjects` returns error `invalid response subject struct`
because `ListSubjects` returns only subject addresses (see frostfs-contract).
Replace `include-names` with `extended` as now all subject info printed.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 13:42:29 +00:00
07a660fbc4
[ #1677 ] writecache: Add QoS limiter usage
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 16:23:33 +03:00
7893d763d1
[ #1673 ] logger: Add sampling for journald logger
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:03 +03:00
ff4e9b6ae1
[ #1673 ] logger: Drop unused fields
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:02 +03:00
997759994a
[ #1676 ] golangci: Enable gci linter
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-13 12:04:01 +03:00
ecb6b0793c
[ #1671 ] Use slices.ContainsFunc() where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
460e5cbccf
[ #1671 ] Use slices.Delete() where possible
...
gopatch is missing for this one, because
https://github.com/uber-go/gopatch/issues/179
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
155d3ddb6e
[ #1671 ] Use min builtin where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
