Commit graph

5440 commits

Author SHA1 Message Date
bd1c18e117
[#1689] cli/tree: Copy dial options from the service code
There should be no `grpcs://` prefix in address and credentials should
be picked.

Change-Id: I58cdc98b079eac2c7db7dc088f4f131794a91b9f
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-04-01 14:40:33 +03:00
b27f7d1d17
[#1689] treesvc: Use context dialer in synchronizeTree()
This dialer supports source-based routing and is already used in cache.

Change-Id: Ic7852edd2faea4e5d8667221e6f681cc82bb143a
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-04-01 14:40:33 +03:00
3cd8080232
[#1689] treesvc: Fix dial options for TLS connections
There are two problems with the current approach:
1. For TLS connections we need different transport credentials.
2. grpc.NewClient() considers scheme from `URIAddr()` as a scheme for a
   resolver. `grpcs://` scheme doesn't exist, though, so the default one
   is picked. The default resolver (`dns://`) is in turn unable to parse the
   5edab9e554/internal/resolver/dns/dns_resolver.go (L405)
   The error  is `grpcs://192.168.198.248:8081:443: too many colons in address`.

Both problems don't exist in the SDK code, take it from there.

Change-Id: Ia1212050f539162a560796685efdc3f9cfbf80a0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-04-01 14:40:33 +03:00
a11b54ca15
[#1689] treesvc: Unify gRPC client creation for cache and sync
They connect to the same endpoints, the only difference is that
connection for synchronization is limited in lifetime and is closed
after the sync is finished. This is probably not intentional, as
synchronization was implemented before cache was introduced.
However, reusing dialTreeService() in sync.go has possible perfomance
implications, so is avoided for now.

Change-Id: I2e37befd783b4d873ff833969f932deded1195be
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-04-01 14:40:33 +03:00
b112a92408
[#1689] treesvc: Create request after client is initialized
Make it easier to follow.

Change-Id: I40c4db77f015bb45cb25f16ce24e68188fc14380
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-04-01 14:40:33 +03:00
19ca907223
[#1689] treesvc: Untie createConnection() from Service struct
Change-Id: I6212de4b81afe8c2516981a7bb2fea099c7df773
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-04-01 14:40:33 +03:00
f62d81e26a
[#1700] gc: Take mode mutex in locks handlers
Change-Id: I4408eae3aed936f85427b6246dcf727bd6813a0d
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-04-01 14:23:03 +03:00
27899598dc
[#1700] gc: Drop Event interface
There is only one event: new epoch.

Change-Id: I982f3650f7bc753ff2782393625452f0f8cdcc35
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-04-01 14:23:02 +03:00
bc6cc9ae2a
[#1700] engine: Print stacks on test request limiter
Change-Id: I4952769ca431d1049955823b41b99b0984b385fc
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-04-01 14:23:02 +03:00
6e1576cfdb [#1656] qos: Add tests for AdjustOutgoingIOTag Interceptors
Change-Id: If534e756b26cf7f202039d48ecdf554b4283728b
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-04-01 11:55:15 +03:00
a5bae6c5af
[#1699] qos: Allow to prohibit operations for IO tag
Change-Id: I2bee26885244e241d224860978b6de3526527e96
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-04-01 10:08:03 +03:00
5a13830a94
[#1699] mod: Bump frostfs-qos version
Change-Id: Ie5e708c0ca653596c6e3346aa286618868a5aee8
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-04-01 10:08:03 +03:00
dcb2b23a7d [#1656] qos: Add test for SetCriticalIOTag Interceptor
Change-Id: I4a55fcb84e6f65408a1c0120ac917e49e23354a1
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-31 18:21:48 +03:00
115aae7c34 [#1656] qos: Add tests for MaxActiveRPCLimiter Interceptors
Change-Id: Ib65890ae5aec34c34e15d4ec1f05952f74f1ad26
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-31 18:21:46 +03:00
12a0537a7a [#1689] ci: Add commit checker to Jenkinsfile
- Commit checker image is built from dco-go:
  TrueCloudLab/dco-go#14
- 'pull_request_target' branch is defined in Jenkins job:
  TrueCloudLab/jenkins#10
  TrueCloudLab/jenkins#11

Change-Id: Ib86c5749f9e084d736b868240c4b47014b02ba8d
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-31 15:08:59 +00:00
30d4692c3e [#1640] go.mod: Bump version for frostfs-locode-db
Change-Id: Ic45ae77d6209c0097575fc8f89b076b22d50d149
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-31 10:29:41 +00:00
2254c8aff5 [#1689] go.mod: Bump SDK version
Change-Id: Ic946aa68c3d6da9e7d54363f8e9141c6547707d6
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-31 11:55:29 +03:00
d432bebef4 [#1689] go.mod: Bump frostfs-qos version
Change-Id: Iaa28da1a1e7b2f4ab7fd8ed661939eb38f4c7782
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-28 12:40:05 +00:00
d144abc977 [#1692] metabase: Remove useless count variable
It is always equal to `len(to)`.

Change-Id: Id7a4c26e9711216b78f96e6b2511efa0773e3471
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:16:01 +00:00
a2053870e2 [#1692] metabase: Use bucket cache in ListWithCursor()
No changes in speed, but unified approach:
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                           │    master    │                 new                 │
                           │    sec/op    │    sec/op     vs base               │
ListWithCursor/1_item-8      6.067µ ±  8%   5.731µ ± 10%       ~ (p=0.052 n=10)
ListWithCursor/10_items-8    25.40µ ± 11%   26.12µ ± 13%       ~ (p=0.971 n=10)
ListWithCursor/100_items-8   210.7µ ±  9%   203.2µ ±  6%       ~ (p=0.280 n=10)
geomean                      31.90µ         31.22µ        -2.16%

                           │    master    │                  new                  │
                           │     B/op     │     B/op      vs base                 │
ListWithCursor/1_item-8      3.287Ki ± 0%   3.287Ki ± 0%       ~ (p=1.000 n=10) ¹
ListWithCursor/10_items-8    15.63Ki ± 0%   15.62Ki ± 0%       ~ (p=0.328 n=10)
ListWithCursor/100_items-8   138.1Ki ± 0%   138.1Ki ± 0%       ~ (p=0.340 n=10)
geomean                      19.21Ki        19.21Ki       -0.00%
¹ all samples are equal

                           │   master    │                 new                  │
                           │  allocs/op  │  allocs/op   vs base                 │
ListWithCursor/1_item-8       109.0 ± 0%    109.0 ± 0%       ~ (p=1.000 n=10) ¹
ListWithCursor/10_items-8     380.0 ± 0%    380.0 ± 0%       ~ (p=1.000 n=10) ¹
ListWithCursor/100_items-8   3.082k ± 0%   3.082k ± 0%       ~ (p=1.000 n=10) ¹
geomean                       503.5         503.5       +0.00%
¹ all samples are equal
```

Change-Id: Ic11673427615053656b2a60068a6d4dbd27af2cb
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:16:01 +00:00
d00c606fee [#652] adm: Group independent stages in batches
Each stage waits until transaction persists. This is needed to ensure
the next stage will see the result of the previous one. However, some of
the stages do not depend one on another, so we may execute them in
parallel.

`AwaitDisabled` flag is used to localize this batching on the code
level. We could've removed `AwaitTx()` from respective stages, but it
seems more error prone.

Close #652.

Change-Id: Ib9c6f6cd5e0db0f31aa1cda8e127b1fad5166336
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:15:21 +00:00
60446bb668 [#1689] adm/helper: Use proper nns bindings import
The one in `neo-go` is for another contract.

Change-Id: Ia1ac2da5e419b48801afdb26df72892d77344e0d
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:15:21 +00:00
bd8ab2d84a [#1689] adm: Remove useless switch in NNSIsAvailable()
After all the refactorings, there is no need to have custom behaviour
for local client.

Change-Id: I99e297cdeffff979524b3f89d3580ab5780e7681
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:15:21 +00:00
bce2f7bef0 [#1689] adm: Reuse neo.NewReader helper in transferNEOFinished()
Change-Id: I27980ed87436958cb4d27278e30e05da021d1506
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:10:37 +00:00
c2c05e2228 [#1689] adm: Reuse ReadOnlyInvoker in registerCandidateRange()
Change-Id: I544d10340825494b45a62700fa247404c18f746a
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:10:37 +00:00
0a38571a10 [#1689] adm: Simplify getCandidateRegisterPrice()
After all the refactoring, there is no more need to have custom branch
for the local client.

Change-Id: I274305b0c390578fb4583759135d3e7ce58873dc
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-28 12:10:31 +00:00
632bd8e38d [#1696] qos: Fix internal tag adjust
If request has no tag, but request's public key is netmap node's key or
one of allowed internal tag keys from config, then request must use
internal IO tag.

Change-Id: Iff93b626941a81b088d8999b3f2947f9501dcdf8
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-28 07:47:12 +00:00
3bbee1b554 [#1619] logger: Allow to set options for zap.Logger via logger.Prm
Change-Id: I8eed951c25d1ecf18b0aea62c6825be65a450085
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-27 15:41:37 +03:00
9358938222 [#1633] go.mod: Bump frostfs-sdk-go
Change-Id: I50c1a0d5b88e307402a5b1b2883bb9b9a357a2c7
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-26 15:03:17 +00:00
5470b205fd [#1619] gc: Fix metric frostfs_node.garbage_collector.marking_duration_seconds
Change-Id: I957f930d1babf179d0fb6de624a90f4fe9977862
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-26 14:14:26 +03:00
163e2e9f83 ci: Cache pre-commit installations on Jenkins Agent
This change introduces a custom helper from our shared library [0]
that defines ad-hoc container environment to execute CI steps in.

Pre-commit installation will now be cached on Jenkins Agent: builds will
tolerate network hiccups better and we will also save some run time
(although on non-critical path of a parallel process).

[0]: TrueCloudLab/jenkins#8

Change-Id: I93b01f169c457aa35f4d8bc5b90f31b31e2bd8b2
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-24 17:04:24 +03:00
0c664fa804 [#1689] qos: Fix metric description
Change-Id: I460fdd3713e765d57ef3ff2945b9b3776f46c164
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-24 09:42:45 +00:00
0a9d139e20 [#1689] morph/client: Remove notary hash field from notaryInfo
Notary contract hash is constant.

Change-Id: I7935580acbced5c9d567875ea75daa57cc259a3c
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-24 09:20:19 +00:00
3bb1fb744a [#1689] morph/client: Reuse auto-generated wrappers for NNS
Make code simpler, remove unused methods.

Change-Id: I18807f2c14b5a96e533e5e3fc153e23c742c66c1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-24 09:20:19 +00:00
ccdd6cb767 [#1052] object: Nuke out acl middleware
* Remove `acl` package as it's no longer used;
* Remove `RequestContext`;
* Fix `cmd/frostfs-node`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-23 06:39:32 +00:00
73e35bc885 [#1052] object: Make ape middleware form request info
* Move some helpers from `acl/v2` package to `ape`. Also move errors;
* Introduce `Metadata`, `RequestInfo` types;
* Introduce `RequestInfoExtractor` interface and its implementation.
  The extractor's purpose is to extract request info based on request
  metadata. It also validates session token;
* Refactor ape service - each handler forms request info and pass
  necessary fields to checker.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-23 06:39:32 +00:00
eed0824590 go.mod: Bump frostfs-qos version
Change-Id: I8bc045b509ee1259cfad288477a0b7d045683f10
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 16:36:03 +00:00
a4da1da767 [#905] morph/client: Fetch NNS hash once on init
NNS contract hash is taken from the contract with ID=1.
Because morph client is expected to work with the same chain,
and because contract hash doesn't change on update, there is no need to
fetch it from each new endpoint.

Change-Id: Ic6dc18283789da076d6a0b3701139b97037714cc
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:13:54 +00:00
30099194ba [#1689] config: Remove testnet and mainnet configs
They are invalid and unsupported.
There is neither mainnet nor testnet currently.

Change-Id: I520363e2de0c22a584238accc253248be3eefea5
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:07:02 +00:00
e7e91ef634 [#1689] adm: Remove storagecfg subcommand
It is unused and unsupported for a long time.

Change-Id: I570567db4e8cb202e41286064406ad85cd0e7a39
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 15:05:02 +00:00
4919b6a206
[#1689] node/config: Allow zero max_ops in RPC limits config
The limiter allows zeros for limits, meaning "this operation is
disabled".  However, the config didn't allow zero due to the lack of
distinction between "no value" and "zero" - cast functions read both
`nil` and zero as zero.

Now, the config allows a zero limit. Added tests.

Managing such cases should be easier after #1610.

Change-Id: Ifc840732390b2feb975f230573b34bf479406e05
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-03-21 17:00:40 +03:00
d951289131 [#1294] docs: Fix description of shard switching mode
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-21 12:13:03 +00:00
016f2e11e3 [#1689] Makefile: Add more restricted .SHELLFLAGS
Catch more errors immediately.

Change-Id: I576f1b394a2b167c78c693a794ab8cca3ac1013b
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:49:49 +00:00
9aa486c9d8 [#1689] Makefile: Create dirs with -p flag
On CI there is no `bin` directory initially, so an error occurs
```
mkdir: cannot create directory '/var/cache/jenkins-agent/workspace/gerrit/frostfs-node#55-488b12-8ac3c/bin/gofumpt': No such file or directory
```

Change-Id: I43895c8f5ed7cc5c71c8025228710279f9e75e9c
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:49:42 +00:00
af76350bfb [#1695] qos: Sort tags by asc
Change-Id: Ia23e392bb49d2536096de2ba07fc6f8fb7ac0489
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 11:25:33 +00:00
3fa5c22ddf [#1689] Makefile: Add default reviewers via --push-option
Gerrit doesn't provide an easy way to have default reviewers assigned to
new change requests. However, we can use `--push-option` and mention all
people from storage-core-developers group.

Change-Id: Ia01f8a3c5c8eb8a1dca6efb66fbe07018f6a42c9
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 11:25:21 +00:00
5385f9994f
[#1695] mod: Bump frostfs-observability version
Change-Id: Id362b71f743ff70c8cd374030c9fa67e2566022f
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:02 +03:00
eea46a599d
[#1695] qos: Add treesync tag
Tree sync is too much different from GC and rebuild to use the same tag for GC and tree sync.

Change-Id: Ib44d5fa9a88daff507d759d0b0410cc9272e236f
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:02 +03:00
049a650b89 [#1619] logger: Simplify logger config reloading
Change-Id: Ide892b250304b8cdb6c279f5f728c3b35f05df54
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-21 09:03:54 +00:00
3f4717a37f [#1692] metabase: Do not allocate map in cache unless needed
Change-Id: I8b1015a8c7c3df4153a08fdb788117d9f0d6c333
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 08:56:32 +00:00