fyrchik/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
5349 commits 20 branches 26 tags 25 MiB
Commit graph

5349 commits

This branch
This branch
All branches
Author SHA1 Message Date
Evgenii Stratonikov
6260d703ce
[#1665] treesvc: Disable service config query 
By default, gRPC fetches TXT report while resolving a domain.
0914bba6c5/internal/resolver/dns/dns_resolver.go (L336)

This leads to a hanging dial if DNS is unavailable, even though the host
may be specified in `/etc/hosts` (hello, localhost!).

SDK client for the main API uses these options by default.

Refs TrueCloudLab/frostfs-sdk-go#342

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-03-06 15:15:31 +03:00
Evgenii Stratonikov
a17c3356fa
[#1665] go.mod: Update sdk-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-03-06 15:15:31 +03:00
Ekaterina Lebedeva
471aeeaff3 [#1659] audit: Fix duplicated request logs 
When we do `object put` with audit enabled we get several entries
in logs: with and without object id.

`object put` request is logged in 2 places:
1. `(*auditPutStream) CloseAndRecv()` - when the client closes the
   request stream or when stream gets aborted.
2. `(*auditPutStream) Send()` - when stream was NOT aborted.

`Send()` does error check for `ErrAbortStream` because if there
is any other error - CloseAndRecv will not be called and there
won't be any audit log about failed request.
It led to logging on every object chunck put, even if `err == nil`.

Added check for `err != nil` in `Send()` to fix it.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-03-06 12:36:16 +03:00
Airat Arifullin
4c8f9580a1 [#1662] object: Fix CloseAndRecv for patch streamer 
* A client may open stream to server, not send anything and close
  the open stream immediatly. This shouldn't cause a panic;
* Return the error if `s.patcher` is uninitialized. Uninitialized
  patcher cannot be closed, this causes a panic.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2025-03-05 17:16:10 +03:00
Aleksey Savchuk
bf8914fedc [#1416] lens/explorer: Support metabase schema v3 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-03-05 07:20:05 +00:00
Evgenii Stratonikov
5ba0e2918e
[#1661] cli: Clarify --rpc-endpoint values 
`multiaddr` is not something an average user knows. Personally, I have
never used it in CLI. On the other hand, we need to connect with TLS quite often,
so it needs to be mentioned in help.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-03-04 10:19:49 +03:00
Dmitrii Stepanov
4685afb1dc
[#1636] engine: Validate limiter release in unit tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:30 +03:00
Dmitrii Stepanov
eb8b9b2b3b
[#1636] blobovniczatree: Validate limiter release in rebuild unit tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:30 +03:00
Dmitrii Stepanov
6c6e463b73
[#1636] shard: Change ops limiter on shard reload 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:29 +03:00
Dmitrii Stepanov
401d96a89e
[#1636] config: Refactor newConfig and oldConfig 
`newConfig` is actually target config to set config values from source
(which is called `oldConfig`).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:29 +03:00
Dmitrii Stepanov
8ed71a969e
[#1636] qos: Add semaphore limiter 
If no tags specified, then limiter could be optimized to use atomic semaphore.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:29 +03:00
Dmitrii Stepanov
c2d855aedd
[#1636] qos: Return Resource Exhausted error 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:28 +03:00
Dmitrii Stepanov
2162f8e189
[#1636] object: Fix IO tag adjustment for Put/Patch 
There was no tag adjustment for CloseAndRecv.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:28 +03:00
Dmitrii Stepanov
b9360be1dc
[#1636] blobovniczatree: Use RebuildLimiter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:27 +03:00
Dmitrii Stepanov
ceff5e1f6a
[#1636] storage: Refactor shard rebuild 
Drop redundant interfaces.
Rename fields.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:27 +03:00
Dmitrii Stepanov
e0dc3c3d0c
[#1636] shard: Add limiter usage 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:26 +03:00
Dmitrii Stepanov
92a67a6716
[#1636] qos: Add Limiter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:26 +03:00
Dmitrii Stepanov
98d6125029
[#1636] config: Add shard.limits config 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 17:25:25 +03:00
Dmitrii Stepanov
0991077cb3 [#1657] engine: Fix data race in evacuation tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-28 14:14:12 +00:00
Aleksey Savchuk
c660271039
[#1658] node: Validate RPC limiter configuration 
Validate that configured limits match the methods registered earlier.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-28 16:31:47 +03:00
Aleksey Savchuk
92ab58984b
[#1658] node: Simplify RPC limiter initialization 
- Move all initialization logic to one place
- Initialize the limiter after all RPC services are registered to be able
  to validate that configured limits match the methods registered earlier

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-28 16:24:33 +03:00
Aleksey Savchuk
dae0949f6e [#1639] node: Support active RPC limiting 
- Allow configuration of active RPC limits for method groups
- Apply RPC limiting for all services except the control service

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-28 11:08:09 +00:00
Aleksey Savchuk
5590886599 [#1639] qos: Add interceptors for limiting active RPCs 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-28 11:08:09 +00:00
Aleksey Savchuk
f0b2017057 [#1639] go.mod: Update sdk-go and qos 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-28 11:08:09 +00:00
Aleksey Savchuk
dce269c62e [#1639] services/object: Remove limiting pools for Put operation 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-28 11:08:09 +00:00
Aleksey Savchuk
a97bded440 [#1639] config: Separate replicator.pool_size from other settings 
Separated `replicator.pool_size` and `object.put.remote_pool_size` settings.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-28 11:08:09 +00:00
Airat Arifullin
9a0507704a [#1651] apemanager: Wrap some errors with InvalidArgument status 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2025-02-27 18:25:09 +00:00
Alexander Chuprov
2ff032db90 [#1654] adm: Get 'gasInitialTotalSupply' from network 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-27 11:57:45 +00:00
Alexander Chuprov
37972a91c1 [#1654] adm: Make 'morph-init' idempotent 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-27 11:57:45 +00:00
Evgenii Stratonikov
003d568ae2
[#1628] innerring: Relax container homomorphic hashing check 
Our initial desire was to prohibit using homomorphic hashing on the
network level because of the resource consumption. However, the ability
to use it, doesn't mean that we must. So only fail validation if
container wants to have homomorphic hashing, but the network prohibits
it.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-02-27 10:20:08 +03:00
Alexander Chuprov
b2adf1109e
[#1646] cli: Use Cmp() functions for ID-like structs 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-20 12:27:07 +03:00
Aleksey Savchuk
02f3a7f65c
[#1648] writecache: Fix race condition when reporting cache size metrics 
There is a race condition when multiple cache operation try to report
the cache size metrics simultaneously. Consider the following example:
- the initial total size of objects stored in the cache size is 2
- worker X deletes an object and reads the cache size, which is 1
- worker Y deletes an object and reads the cache size, which is 0
- worker Y reports the cache size it learnt, which is 0
- worker X reports the cache size it learnt, which is 1

As a result, the observed cache size is 1 (i. e. one object remains
in the cache), which is incorrect because the actual cache size is 0.

To fix this, let's report the metrics periodically in the flush loop.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-19 17:05:40 +03:00
Dmitrii Stepanov
9b29e7392f
[#1647] go.mod: Bump frostfs-sdk-go version 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-18 09:39:43 +03:00
Dmitrii Stepanov
fe0cf86dc6
[#1643] go.mod: Bump frostfs-observability version 
To add `grpc_client_msg_send_handling_seconds` metric.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-12 15:39:29 +03:00
Vitaliy Potyarkin
1bcaa1af1f [#1641] govulncheck: Fix minor toolchain updates for good 
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
 2025-02-11 14:02:23 +00:00
Alexander Chuprov
304bee938b
[#1614] adm/nns: Add 'set-admin' 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-11 12:00:51 +03:00
Alexander Chuprov
b2163ff44c
[#1614] adm: Allow use any wallets 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-11 12:00:25 +03:00
Alexander Chuprov
076952f4c7
[#1614] adm/frostfsid: Add 'delete-kv' 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-10 15:28:29 +03:00
Alexander Chuprov
a7145ca9bf
[#1614] adm/frostfsid: Add 'set-kv' 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-10 15:26:12 +03:00
Dmitrii Stepanov
5d79abe523
[#1608] shard: Add IO tag for rebuild 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:07 +03:00
Dmitrii Stepanov
0671c277db
[#1608] tree: Add IO tag for tree sync requests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:06 +03:00
Dmitrii Stepanov
92450a76ba
[#1608] shard: Add IO tag to GC 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:06 +03:00
Dmitrii Stepanov
abba5b2089
[#1608] writecache: Add IO tag to flush worker 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:06 +03:00
Dmitrii Stepanov
fd0c6c461d
[#1608] cli: Drop redundant interceptors 
There is no metrics or IO tags in cli.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:05 +03:00
Dmitrii Stepanov
bfe325e61d
[#1608] policer: Add IO tag to context 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:05 +03:00
Dmitrii Stepanov
acec938b2d
[#1608] qos: Add client grpc interceptors 
`qos` client interceptors replace internal IO tags `writecache`,
`policer` and `background` with `internal` IO tag for outcomming RPC.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:04 +03:00
Dmitrii Stepanov
dc6aea7b79
[#1608] control: Use IO tag critical 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:04 +03:00
Dmitrii Stepanov
170860c14a
[#1608] logger: Add IO tag logging 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:03 +03:00
Dmitrii Stepanov
12da2f8262
[#1608] object: Add IO tag adjustment layer 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:03 +03:00
Dmitrii Stepanov
f6b3f79e89
[#1608] qos: Add qos service to adjust incoming IO tags 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:03 +03:00