Commit graph

5432 commits

Author SHA1 Message Date
bc6cc9ae2a
[#1700] engine: Print stacks on test request limiter
Change-Id: I4952769ca431d1049955823b41b99b0984b385fc
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-04-01 14:23:02 +03:00
6e1576cfdb [#1656] qos: Add tests for AdjustOutgoingIOTag Interceptors
Change-Id: If534e756b26cf7f202039d48ecdf554b4283728b
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-04-01 11:55:15 +03:00
a5bae6c5af
[#1699] qos: Allow to prohibit operations for IO tag
Change-Id: I2bee26885244e241d224860978b6de3526527e96
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-04-01 10:08:03 +03:00
5a13830a94
[#1699] mod: Bump frostfs-qos version
Change-Id: Ie5e708c0ca653596c6e3346aa286618868a5aee8
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-04-01 10:08:03 +03:00
dcb2b23a7d [#1656] qos: Add test for SetCriticalIOTag Interceptor
Change-Id: I4a55fcb84e6f65408a1c0120ac917e49e23354a1
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-31 18:21:48 +03:00
115aae7c34 [#1656] qos: Add tests for MaxActiveRPCLimiter Interceptors
Change-Id: Ib65890ae5aec34c34e15d4ec1f05952f74f1ad26
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-31 18:21:46 +03:00
12a0537a7a [#1689] ci: Add commit checker to Jenkinsfile
- Commit checker image is built from dco-go:
  TrueCloudLab/dco-go#14
- 'pull_request_target' branch is defined in Jenkins job:
  TrueCloudLab/jenkins#10
  TrueCloudLab/jenkins#11

Change-Id: Ib86c5749f9e084d736b868240c4b47014b02ba8d
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-31 15:08:59 +00:00
30d4692c3e [#1640] go.mod: Bump version for frostfs-locode-db
Change-Id: Ic45ae77d6209c0097575fc8f89b076b22d50d149
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-31 10:29:41 +00:00
2254c8aff5 [#1689] go.mod: Bump SDK version
Change-Id: Ic946aa68c3d6da9e7d54363f8e9141c6547707d6
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-31 11:55:29 +03:00
d432bebef4 [#1689] go.mod: Bump frostfs-qos version
Change-Id: Iaa28da1a1e7b2f4ab7fd8ed661939eb38f4c7782
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-28 12:40:05 +00:00
d144abc977 [#1692] metabase: Remove useless count variable
It is always equal to `len(to)`.

Change-Id: Id7a4c26e9711216b78f96e6b2511efa0773e3471
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:16:01 +00:00
a2053870e2 [#1692] metabase: Use bucket cache in ListWithCursor()
No changes in speed, but unified approach:
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                           │    master    │                 new                 │
                           │    sec/op    │    sec/op     vs base               │
ListWithCursor/1_item-8      6.067µ ±  8%   5.731µ ± 10%       ~ (p=0.052 n=10)
ListWithCursor/10_items-8    25.40µ ± 11%   26.12µ ± 13%       ~ (p=0.971 n=10)
ListWithCursor/100_items-8   210.7µ ±  9%   203.2µ ±  6%       ~ (p=0.280 n=10)
geomean                      31.90µ         31.22µ        -2.16%

                           │    master    │                  new                  │
                           │     B/op     │     B/op      vs base                 │
ListWithCursor/1_item-8      3.287Ki ± 0%   3.287Ki ± 0%       ~ (p=1.000 n=10) ¹
ListWithCursor/10_items-8    15.63Ki ± 0%   15.62Ki ± 0%       ~ (p=0.328 n=10)
ListWithCursor/100_items-8   138.1Ki ± 0%   138.1Ki ± 0%       ~ (p=0.340 n=10)
geomean                      19.21Ki        19.21Ki       -0.00%
¹ all samples are equal

                           │   master    │                 new                  │
                           │  allocs/op  │  allocs/op   vs base                 │
ListWithCursor/1_item-8       109.0 ± 0%    109.0 ± 0%       ~ (p=1.000 n=10) ¹
ListWithCursor/10_items-8     380.0 ± 0%    380.0 ± 0%       ~ (p=1.000 n=10) ¹
ListWithCursor/100_items-8   3.082k ± 0%   3.082k ± 0%       ~ (p=1.000 n=10) ¹
geomean                       503.5         503.5       +0.00%
¹ all samples are equal
```

Change-Id: Ic11673427615053656b2a60068a6d4dbd27af2cb
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:16:01 +00:00
d00c606fee [#652] adm: Group independent stages in batches
Each stage waits until transaction persists. This is needed to ensure
the next stage will see the result of the previous one. However, some of
the stages do not depend one on another, so we may execute them in
parallel.

`AwaitDisabled` flag is used to localize this batching on the code
level. We could've removed `AwaitTx()` from respective stages, but it
seems more error prone.

Close #652.

Change-Id: Ib9c6f6cd5e0db0f31aa1cda8e127b1fad5166336
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:15:21 +00:00
60446bb668 [#1689] adm/helper: Use proper nns bindings import
The one in `neo-go` is for another contract.

Change-Id: Ia1ac2da5e419b48801afdb26df72892d77344e0d
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:15:21 +00:00
bd8ab2d84a [#1689] adm: Remove useless switch in NNSIsAvailable()
After all the refactorings, there is no need to have custom behaviour
for local client.

Change-Id: I99e297cdeffff979524b3f89d3580ab5780e7681
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:15:21 +00:00
bce2f7bef0 [#1689] adm: Reuse neo.NewReader helper in transferNEOFinished()
Change-Id: I27980ed87436958cb4d27278e30e05da021d1506
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:10:37 +00:00
c2c05e2228 [#1689] adm: Reuse ReadOnlyInvoker in registerCandidateRange()
Change-Id: I544d10340825494b45a62700fa247404c18f746a
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:10:37 +00:00
0a38571a10 [#1689] adm: Simplify getCandidateRegisterPrice()
After all the refactoring, there is no more need to have custom branch
for the local client.

Change-Id: I274305b0c390578fb4583759135d3e7ce58873dc
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:10:31 +00:00
632bd8e38d [#1696] qos: Fix internal tag adjust
If request has no tag, but request's public key is netmap node's key or
one of allowed internal tag keys from config, then request must use
internal IO tag.

Change-Id: Iff93b626941a81b088d8999b3f2947f9501dcdf8
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-28 07:47:12 +00:00
3bbee1b554 [#1619] logger: Allow to set options for zap.Logger via logger.Prm
Change-Id: I8eed951c25d1ecf18b0aea62c6825be65a450085
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-27 15:41:37 +03:00
9358938222 [#1633] go.mod: Bump frostfs-sdk-go
Change-Id: I50c1a0d5b88e307402a5b1b2883bb9b9a357a2c7
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-26 15:03:17 +00:00
5470b205fd [#1619] gc: Fix metric frostfs_node.garbage_collector.marking_duration_seconds
Change-Id: I957f930d1babf179d0fb6de624a90f4fe9977862
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-26 14:14:26 +03:00
163e2e9f83 ci: Cache pre-commit installations on Jenkins Agent
This change introduces a custom helper from our shared library [0]
that defines ad-hoc container environment to execute CI steps in.

Pre-commit installation will now be cached on Jenkins Agent: builds will
tolerate network hiccups better and we will also save some run time
(although on non-critical path of a parallel process).

[0]: TrueCloudLab/jenkins#8

Change-Id: I93b01f169c457aa35f4d8bc5b90f31b31e2bd8b2
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-24 17:04:24 +03:00
0c664fa804 [#1689] qos: Fix metric description
Change-Id: I460fdd3713e765d57ef3ff2945b9b3776f46c164
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-24 09:42:45 +00:00
0a9d139e20 [#1689] morph/client: Remove notary hash field from notaryInfo
Notary contract hash is constant.

Change-Id: I7935580acbced5c9d567875ea75daa57cc259a3c
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-24 09:20:19 +00:00
3bb1fb744a [#1689] morph/client: Reuse auto-generated wrappers for NNS
Make code simpler, remove unused methods.

Change-Id: I18807f2c14b5a96e533e5e3fc153e23c742c66c1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-24 09:20:19 +00:00
ccdd6cb767 [#1052] object: Nuke out acl middleware
* Remove `acl` package as it's no longer used;
* Remove `RequestContext`;
* Fix `cmd/frostfs-node`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-23 06:39:32 +00:00
73e35bc885 [#1052] object: Make ape middleware form request info
* Move some helpers from `acl/v2` package to `ape`. Also move errors;
* Introduce `Metadata`, `RequestInfo` types;
* Introduce `RequestInfoExtractor` interface and its implementation.
  The extractor's purpose is to extract request info based on request
  metadata. It also validates session token;
* Refactor ape service - each handler forms request info and pass
  necessary fields to checker.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-23 06:39:32 +00:00
eed0824590 go.mod: Bump frostfs-qos version
Change-Id: I8bc045b509ee1259cfad288477a0b7d045683f10
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 16:36:03 +00:00
a4da1da767 [#905] morph/client: Fetch NNS hash once on init
NNS contract hash is taken from the contract with ID=1.
Because morph client is expected to work with the same chain,
and because contract hash doesn't change on update, there is no need to
fetch it from each new endpoint.

Change-Id: Ic6dc18283789da076d6a0b3701139b97037714cc
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:13:54 +00:00
30099194ba [#1689] config: Remove testnet and mainnet configs
They are invalid and unsupported.
There is neither mainnet nor testnet currently.

Change-Id: I520363e2de0c22a584238accc253248be3eefea5
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:07:02 +00:00
e7e91ef634 [#1689] adm: Remove storagecfg subcommand
It is unused and unsupported for a long time.

Change-Id: I570567db4e8cb202e41286064406ad85cd0e7a39
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:05:02 +00:00
4919b6a206
[#1689] node/config: Allow zero max_ops in RPC limits config
The limiter allows zeros for limits, meaning "this operation is
disabled".  However, the config didn't allow zero due to the lack of
distinction between "no value" and "zero" - cast functions read both
`nil` and zero as zero.

Now, the config allows a zero limit. Added tests.

Managing such cases should be easier after #1610.

Change-Id: Ifc840732390b2feb975f230573b34bf479406e05
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-03-21 17:00:40 +03:00
d951289131 [#1294] docs: Fix description of shard switching mode
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-21 12:13:03 +00:00
016f2e11e3 [#1689] Makefile: Add more restricted .SHELLFLAGS
Catch more errors immediately.

Change-Id: I576f1b394a2b167c78c693a794ab8cca3ac1013b
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:49:49 +00:00
9aa486c9d8 [#1689] Makefile: Create dirs with -p flag
On CI there is no `bin` directory initially, so an error occurs
```
mkdir: cannot create directory '/var/cache/jenkins-agent/workspace/gerrit/frostfs-node#55-488b12-8ac3c/bin/gofumpt': No such file or directory
```

Change-Id: I43895c8f5ed7cc5c71c8025228710279f9e75e9c
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:49:42 +00:00
af76350bfb [#1695] qos: Sort tags by asc
Change-Id: Ia23e392bb49d2536096de2ba07fc6f8fb7ac0489
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 11:25:33 +00:00
3fa5c22ddf [#1689] Makefile: Add default reviewers via --push-option
Gerrit doesn't provide an easy way to have default reviewers assigned to
new change requests. However, we can use `--push-option` and mention all
people from storage-core-developers group.

Change-Id: Ia01f8a3c5c8eb8a1dca6efb66fbe07018f6a42c9
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:25:21 +00:00
5385f9994f
[#1695] mod: Bump frostfs-observability version
Change-Id: Id362b71f743ff70c8cd374030c9fa67e2566022f
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:02 +03:00
eea46a599d
[#1695] qos: Add treesync tag
Tree sync is too much different from GC and rebuild to use the same tag for GC and tree sync.

Change-Id: Ib44d5fa9a88daff507d759d0b0410cc9272e236f
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:02 +03:00
049a650b89 [#1619] logger: Simplify logger config reloading
Change-Id: Ide892b250304b8cdb6c279f5f728c3b35f05df54
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-21 09:03:54 +00:00
3f4717a37f [#1692] metabase: Do not allocate map in cache unless needed
Change-Id: I8b1015a8c7c3df4153a08fdb788117d9f0d6c333
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 08:56:32 +00:00
60cea8c714 [#1692] metabase/test: Fix end of iteration error check
This is not good:
```
BenchmarkListWithCursor/1_item-8                --- FAIL: BenchmarkListWithCursor/1_item-8
    list_test.go:63: error: end of object listing
```

Change-Id: I61b70937ce30fefaf16ebeb0cdb51bdd39096061
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 08:56:18 +00:00
7df2912a83
[#1689] Makefile: Create prepare-commit-msg hook too
`commit-msg` is ignored when `--no-verify` option is used, so there is
no way to ignore `pre-commit` while retaining `commit-msg` hook.
Ignoring pre-commit is useful, though, so we might add Change-Id in
`prepare-commit-msg` hook instead. It accepts more parameters, but the
first one is a file with the commit message, so we may reuse
`commit-msg` hook.

Change-Id: I4edb79810bbe38a5dcf7f4f07535f34c6bda0da3
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 18:01:53 +03:00
affab25512
Makefile: Add Gerrit-related targets
This commit adds helper targets to easily setup an existing repo for
work with Gerrit.

Change-Id: I0696eb8ea84cc16a9482be6a2fb0382fe624bb96
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 16:48:04 +03:00
45b7796151 [#1689] ci: Reimplement CI tasks in Jenkinsfile
This commit introduces Jenkins pipeline that duplicates the features of
existing Forgejo Actions workflows.

Change-Id: I657a6c27373a1ed4736ae27b4fb660e0ac86012d
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-20 12:55:30 +00:00
e8801dbf49 [#1691] metabase: Move cheaper conditions to the front in ListWithCursor()
`objectLocked` call is expensive, it does IO. We may omit it if object
is not expired.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:36 +00:00
eb9df85b98 [#1685] metabase: Cache primary bucket
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                          │   expired    │               primary               │
                          │    sec/op    │    sec/op     vs base               │
Select/string_equal-8       3.529m ± 11%   3.689m ±  7%  +4.55% (p=0.023 n=10)
Select/string_not_equal-8   3.440m ±  7%   3.543m ± 13%       ~ (p=0.190 n=10)
Select/common_prefix-8      3.240m ±  6%   3.050m ±  5%  -5.85% (p=0.005 n=10)
Select/unknown-8            3.198m ±  6%   2.928m ±  8%  -8.44% (p=0.003 n=10)
geomean                     3.349m         3.287m        -1.84%

                          │   expired    │               primary               │
                          │     B/op     │     B/op      vs base               │
Select/string_equal-8       1.885Mi ± 0%   1.786Mi ± 0%  -5.23% (p=0.000 n=10)
Select/string_not_equal-8   1.885Mi ± 0%   1.786Mi ± 0%  -5.23% (p=0.000 n=10)
Select/common_prefix-8      1.885Mi ± 0%   1.786Mi ± 0%  -5.23% (p=0.000 n=10)
Select/unknown-8            1.877Mi ± 0%   1.779Mi ± 0%  -5.26% (p=0.000 n=10)
geomean                     1.883Mi        1.784Mi       -5.24%

                          │   expired   │              primary               │
                          │  allocs/op  │  allocs/op   vs base               │
Select/string_equal-8       46.04k ± 0%   43.04k ± 0%  -6.50% (p=0.000 n=10)
Select/string_not_equal-8   46.04k ± 0%   43.04k ± 0%  -6.50% (p=0.000 n=10)
Select/common_prefix-8      46.04k ± 0%   43.04k ± 0%  -6.50% (p=0.000 n=10)
Select/unknown-8            45.05k ± 0%   42.05k ± 0%  -6.65% (p=0.000 n=10)
geomean                     45.79k        42.79k       -6.54%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:01 +00:00
21bed3362c [#1685] metabase: Cache expired bucket
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                          │    master    │               expired                │
                          │    sec/op    │    sec/op     vs base                │
Select/string_equal-8       4.007m ± 10%   3.529m ± 11%  -11.94% (p=0.000 n=10)
Select/string_not_equal-8   3.834m ± 12%   3.440m ±  7%  -10.29% (p=0.029 n=10)
Select/common_prefix-8      3.470m ±  9%   3.240m ±  6%        ~ (p=0.105 n=10)
Select/unknown-8            3.156m ±  3%   3.198m ±  6%        ~ (p=0.631 n=10)
geomean                     3.602m         3.349m         -7.03%

                          │    master    │               expired               │
                          │     B/op     │     B/op      vs base               │
Select/string_equal-8       1.907Mi ± 0%   1.885Mi ± 0%  -1.18% (p=0.000 n=10)
Select/string_not_equal-8   1.907Mi ± 0%   1.885Mi ± 0%  -1.18% (p=0.000 n=10)
Select/common_prefix-8      1.907Mi ± 0%   1.885Mi ± 0%  -1.18% (p=0.000 n=10)
Select/unknown-8            1.900Mi ± 0%   1.877Mi ± 0%  -1.18% (p=0.000 n=10)
geomean                     1.905Mi        1.883Mi       -1.18%

                          │   master    │              expired               │
                          │  allocs/op  │  allocs/op   vs base               │
Select/string_equal-8       47.03k ± 0%   46.04k ± 0%  -2.12% (p=0.000 n=10)
Select/string_not_equal-8   47.03k ± 0%   46.04k ± 0%  -2.12% (p=0.000 n=10)
Select/common_prefix-8      47.03k ± 0%   46.04k ± 0%  -2.12% (p=0.000 n=10)
Select/unknown-8            46.04k ± 0%   45.05k ± 0%  -2.16% (p=0.000 n=10)
geomean                     46.78k        45.79k       -2.13%
```

Change-Id: I9c7a5e1f5c8b9eb3f25a563fd74c6ad2a9d1b92e
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:01 +00:00
af5b3575d0
[#1690] qos: Do not export zero metrics counters
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-20 14:42:35 +03:00