Commit graph

5370 commits

Author SHA1 Message Date
07a660fbc4
[#1677] writecache: Add QoS limiter usage
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 16:23:33 +03:00
7893d763d1
[#1673] logger: Add sampling for journald logger
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:03 +03:00
ff4e9b6ae1
[#1673] logger: Drop unused fields
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:02 +03:00
997759994a
[#1676] golangci: Enable gci linter
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-13 12:04:01 +03:00
ecb6b0793c [#1671] Use slices.ContainsFunc() where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
460e5cbccf [#1671] Use slices.Delete() where possible
gopatch is missing for this one, because
https://github.com/uber-go/gopatch/issues/179

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
155d3ddb6e [#1671] Use min builtin where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
40536d8a06 [#1671] Use fmt.Appendf where warranted
Fix gopls warnings:
```
cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf
````

gopatch:
```
@@
var f expression
@@
-[]byte(fmt.Sprintf(f, ...))
+fmt.Appendf(nil, f, ...)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
d66bffb191 [#1671] Use max builtin where possible
gopatcH:
```
@@
var d, a expression
@@
-if d < a {
-    d = a
-}
-return d
+return max(d, a)

@@
var d, a expression
@@
-if d <= a {
-    d = a
-}
-return d
+return max(d, a)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
bcc84c85a0 [#1671] Replace interface{} with any
gopatch:
```
@@
@@
-interface{}
+any
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
737788b35f
[#1669] go.mod: Bump frostfs-qos version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-12 09:23:53 +03:00
2005fdda09
[#1667] shard: Drop shard pool
After adding an ops limiter, shard's `put` pool is redundant.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 13:59:51 +03:00
597bce7a87 [#1653] treeSvc: Add operations by IO tag metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
4ed2bbdb0f [#1653] objectSvc: Add operations by IO tag metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
3727d60331 [#1653] qos: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
d36afa31c7 [#1653] qos: Fix logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
8643e0abc5
[#1668] writecache: Use object size to check free space
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-10 17:52:57 +03:00
bd61f7bf0a [#1666] audit: Fix duplicated log in Patch method
When we do `object patch` with audit enabled we get several
duplicated entries in logs.

`object patch` request is logged in 2 places:
1. `(*auditPatchStream) CloseAndRecv()` - when the client closes
   the request stream or when stream gets aborted.
2. `(*auditPatchStream) Send()` - when stream was NOT aborted.

`Send()` doesn't check if `err != nil` before logging.
It led to to logging on every `Send()` call.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 13:27:07 +00:00
df6d9da82a [#1635] cli: Add command to get object's shard info
Added `frostfs-cli object locate` subcommand. It lists info
about shards storing an object.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
aab8addae0 [#1635] cli: Make object.readObjectAddress() public
This method will be useful for upcoming control command.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
9e31cb249f [#1635] control: Add method to search shards by object
Added method `ListShardsForObject` to ControlService and to
StorageEngine. It returns information about shards storing
object on the node.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
6260d703ce
[#1665] treesvc: Disable service config query
By default, gRPC fetches TXT report while resolving a domain.
0914bba6c5/internal/resolver/dns/dns_resolver.go (L336)

This leads to a hanging dial if DNS is unavailable, even though the host
may be specified in `/etc/hosts` (hello, localhost!).

SDK client for the main API uses these options by default.

Refs TrueCloudLab/frostfs-sdk-go#342

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-06 15:15:31 +03:00
a17c3356fa
[#1665] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-06 15:15:31 +03:00
471aeeaff3 [#1659] audit: Fix duplicated request logs
When we do `object put` with audit enabled we get several entries
in logs: with and without object id.

`object put` request is logged in 2 places:
1. `(*auditPutStream) CloseAndRecv()` - when the client closes the
   request stream or when stream gets aborted.
2. `(*auditPutStream) Send()` - when stream was NOT aborted.

`Send()` does error check for `ErrAbortStream` because if there
is any other error - CloseAndRecv will not be called and there
won't be any audit log about failed request.
It led to logging on every object chunck put, even if `err == nil`.

Added check for `err != nil` in `Send()` to fix it.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-06 12:36:16 +03:00
4c8f9580a1 [#1662] object: Fix CloseAndRecv for patch streamer
* A client may open stream to server, not send anything and close
  the open stream immediatly. This shouldn't cause a panic;
* Return the error if `s.patcher` is uninitialized. Uninitialized
  patcher cannot be closed, this causes a panic.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-05 17:16:10 +03:00
bf8914fedc [#1416] lens/explorer: Support metabase schema v3
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-03-05 07:20:05 +00:00
5ba0e2918e
[#1661] cli: Clarify --rpc-endpoint values
`multiaddr` is not something an average user knows. Personally, I have
never used it in CLI. On the other hand, we need to connect with TLS quite often,
so it needs to be mentioned in help.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-04 10:19:49 +03:00
4685afb1dc
[#1636] engine: Validate limiter release in unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:30 +03:00
eb8b9b2b3b
[#1636] blobovniczatree: Validate limiter release in rebuild unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:30 +03:00
6c6e463b73
[#1636] shard: Change ops limiter on shard reload
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:29 +03:00
401d96a89e
[#1636] config: Refactor newConfig and oldConfig
`newConfig` is actually target config to set config values from source
(which is called `oldConfig`).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:29 +03:00
8ed71a969e
[#1636] qos: Add semaphore limiter
If no tags specified, then limiter could be optimized to use atomic semaphore.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:29 +03:00
c2d855aedd
[#1636] qos: Return Resource Exhausted error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:28 +03:00
2162f8e189
[#1636] object: Fix IO tag adjustment for Put/Patch
There was no tag adjustment for CloseAndRecv.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:28 +03:00
b9360be1dc
[#1636] blobovniczatree: Use RebuildLimiter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:27 +03:00
ceff5e1f6a
[#1636] storage: Refactor shard rebuild
Drop redundant interfaces.
Rename fields.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:27 +03:00
e0dc3c3d0c
[#1636] shard: Add limiter usage
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:26 +03:00
92a67a6716
[#1636] qos: Add Limiter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:26 +03:00
98d6125029
[#1636] config: Add shard.limits config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:25 +03:00
0991077cb3 [#1657] engine: Fix data race in evacuation tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 14:14:12 +00:00
c660271039
[#1658] node: Validate RPC limiter configuration
Validate that configured limits match the methods registered earlier.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-28 16:31:47 +03:00
92ab58984b
[#1658] node: Simplify RPC limiter initialization
- Move all initialization logic to one place
- Initialize the limiter after all RPC services are registered to be able
  to validate that configured limits match the methods registered earlier

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-28 16:24:33 +03:00
dae0949f6e [#1639] node: Support active RPC limiting
- Allow configuration of active RPC limits for method groups
- Apply RPC limiting for all services except the control service

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-28 11:08:09 +00:00
5590886599 [#1639] qos: Add interceptors for limiting active RPCs
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-28 11:08:09 +00:00
f0b2017057 [#1639] go.mod: Update sdk-go and qos
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-28 11:08:09 +00:00
dce269c62e [#1639] services/object: Remove limiting pools for Put operation
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-28 11:08:09 +00:00
a97bded440 [#1639] config: Separate replicator.pool_size from other settings
Separated `replicator.pool_size` and `object.put.remote_pool_size` settings.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-28 11:08:09 +00:00
9a0507704a [#1651] apemanager: Wrap some errors with InvalidArgument status
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-02-27 18:25:09 +00:00
2ff032db90 [#1654] adm: Get 'gasInitialTotalSupply' from network
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-02-27 11:57:45 +00:00
37972a91c1 [#1654] adm: Make 'morph-init' idempotent
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-02-27 11:57:45 +00:00