elebedeva/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Pull requests Activity
5321 commits 26 branches 26 tags 21 MiB
Commit graph

5321 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ekaterina Lebedeva
fbfb2d23d8 [#1635] cli: Add command to get object's shard info 
Added `frostfs-cli object locate` subcommand. It lists info
about shards storing an object.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-02-20 15:11:18 +03:00
Ekaterina Lebedeva
13e2a18d46 [#1635] cli: Make object.readObjectAddress() public 
This method will be useful for upcoming control command.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-02-20 15:11:18 +03:00
Ekaterina Lebedeva
0a4405f9d9 [#1635] control: Add method to search shards by object 
Added method `ListShardsForObject` to ControlService and to
StorageEngine. It returns information about shards storing
object on the node.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-02-20 15:11:18 +03:00
Aleksey Savchuk
02f3a7f65c
[#1648] writecache: Fix race condition when reporting cache size metrics 
There is a race condition when multiple cache operation try to report
the cache size metrics simultaneously. Consider the following example:
- the initial total size of objects stored in the cache size is 2
- worker X deletes an object and reads the cache size, which is 1
- worker Y deletes an object and reads the cache size, which is 0
- worker Y reports the cache size it learnt, which is 0
- worker X reports the cache size it learnt, which is 1

As a result, the observed cache size is 1 (i. e. one object remains
in the cache), which is incorrect because the actual cache size is 0.

To fix this, let's report the metrics periodically in the flush loop.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-19 17:05:40 +03:00
Dmitrii Stepanov
9b29e7392f
[#1647] go.mod: Bump frostfs-sdk-go version 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-18 09:39:43 +03:00
Dmitrii Stepanov
fe0cf86dc6
[#1643] go.mod: Bump frostfs-observability version 
To add `grpc_client_msg_send_handling_seconds` metric.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-12 15:39:29 +03:00
Vitaliy Potyarkin
1bcaa1af1f [#1641] govulncheck: Fix minor toolchain updates for good 
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
 2025-02-11 14:02:23 +00:00
Alexander Chuprov
304bee938b
[#1614] adm/nns: Add 'set-admin' 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-11 12:00:51 +03:00
Alexander Chuprov
b2163ff44c
[#1614] adm: Allow use any wallets 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-11 12:00:25 +03:00
Alexander Chuprov
076952f4c7
[#1614] adm/frostfsid: Add 'delete-kv' 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-10 15:28:29 +03:00
Alexander Chuprov
a7145ca9bf
[#1614] adm/frostfsid: Add 'set-kv' 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-10 15:26:12 +03:00
Dmitrii Stepanov
5d79abe523
[#1608] shard: Add IO tag for rebuild 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:07 +03:00
Dmitrii Stepanov
0671c277db
[#1608] tree: Add IO tag for tree sync requests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:06 +03:00
Dmitrii Stepanov
92450a76ba
[#1608] shard: Add IO tag to GC 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:06 +03:00
Dmitrii Stepanov
abba5b2089
[#1608] writecache: Add IO tag to flush worker 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:06 +03:00
Dmitrii Stepanov
fd0c6c461d
[#1608] cli: Drop redundant interceptors 
There is no metrics or IO tags in cli.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:05 +03:00
Dmitrii Stepanov
bfe325e61d
[#1608] policer: Add IO tag to context 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:05 +03:00
Dmitrii Stepanov
acec938b2d
[#1608] qos: Add client grpc interceptors 
`qos` client interceptors replace internal IO tags `writecache`,
`policer` and `background` with `internal` IO tag for outcomming RPC.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:04 +03:00
Dmitrii Stepanov
dc6aea7b79
[#1608] control: Use IO tag critical 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:04 +03:00
Dmitrii Stepanov
170860c14a
[#1608] logger: Add IO tag logging 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:03 +03:00
Dmitrii Stepanov
12da2f8262
[#1608] object: Add IO tag adjustment layer 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:03 +03:00
Dmitrii Stepanov
f6b3f79e89
[#1608] qos: Add qos service to adjust incoming IO tags 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:43:03 +03:00
Dmitrii Stepanov
9729f31e5c
[#1608] grpc: Add QoS interceptors for server and clients 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:41:06 +03:00
Dmitrii Stepanov
155f9eecb0
[#1608] config: Add QoS section and config 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-07 15:41:06 +03:00
Alexander Chuprov
69c35b1d61
[#1637] govulncheck: Use patch release with security fixes 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-06 21:19:24 +03:00
Alexander Chuprov
9b113c3156
[#1613] morph: Add tracing for morph queries to neo-go 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-02-05 16:38:20 +03:00
Dmitrii Stepanov
4de5fca547
[#1624] blobovniczatree: Return source error from Init() 
As it was before: when the database was opened, an error returned,
but along with the original error, the `context cancelled`` error returned,
because `iterateIncompletedRebuildDBPaths` method has `ctx.Done()` check
and egCtx passed to `iterateIncompletedRebuildDBPaths` method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-02-03 17:36:21 +03:00
Aleksey Savchuk
9c4c5a5262
[#1602] go.mod: Update sdk-go 
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
 2025-02-03 14:28:23 +03:00
Evgenii Stratonikov
6fcae9f75a [#1621] treesvc: Cancel background sync on failure 
If applyOperationStream() exits prematurely, other goroutines will block
on send and errgroup will never finish waiting. In this commit we also
check whether context is cancelled.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-02-03 09:37:55 +00:00
Evgenii Stratonikov
1df64c5cab [#1621] treesvc: Move variable initialization to top in mergeOperationStreams() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-02-03 09:37:55 +00:00
Dmitrii Stepanov
6a580db55e
[#1626] go.mod: Bump frostfs-sdk-go 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-01-30 16:23:00 +03:00
Ekaterina Lebedeva
24054cf6f4 [#1618] pkg: Refactor user.ID.ScriptHash() usage 
`user.ID.ScriptHash()` does not return an error anymore.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-01-30 13:18:37 +00:00
Ekaterina Lebedeva
9ee3dd4e91 [#1618] ape: Fix object service request tests 
Constant string `testOwnerID` for these tests has an invalid
format. It has 11 bytes instead of required 25 for `user.ID`.
It worked because:
  1. `user.ID` was a byte slice and didn't check length
     and format of byte slices decoded from strings.
  2. in these tests `testOwnerID` was used only to decode
     container owner id and to compare it with owner id encoded
     back to string.
Since `user.ID implementation has changed`, the problem arised.
Now `testOwnerID` is valid.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-01-30 13:18:37 +00:00
Dmitrii Stepanov
78bfd12229
[#1622] adm: Return non zero return code if metabase upgrade failed 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-01-30 15:31:25 +03:00
Evgenii Stratonikov
57dc0a8e9e [#1616] getsvc: Move break condition from body to the loop condition 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-30 06:50:37 +00:00
Evgenii Stratonikov
b309b34bfc [#1616] getsvc: Simplify buildChainRange() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-30 06:50:37 +00:00
Evgenii Stratonikov
c8acdf40bb [#1616] getsvc: Use slices.Reverse() where possible 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-30 06:50:37 +00:00
Evgenii Stratonikov
6410542d19 [#1616] getsvc: Move range assembling to a separate file 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-30 06:50:37 +00:00
Evgenii Stratonikov
c0a341a7f6 [#1616] getsvc: Move head assembling to a separate file 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-30 06:50:37 +00:00
Dmitrii Stepanov
e1a984e9d8
[#1620] logs: Drop redundant trace_id fields 
`trace_id` is taken from context.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2025-01-29 16:13:51 +03:00
Vitaliy Potyarkin
abfd9657f9 [#1617] govulncheck: Use patch release with security fixes 
https://go.dev/doc/devel/release#go1.23.minor

Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
 2025-01-28 18:02:57 +03:00
Ekaterina Lebedeva
a788d44773 [#1570] cli: Use array type for attributes parameters 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-01-28 11:15:30 +03:00
Ekaterina Lebedeva
603015d029 [#1570] cli: Use array type for --range parameter to object hash 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2025-01-28 11:15:30 +03:00
Alexander Chuprov
30e14d50ef
[#1612] Makefile: Update golangci-lint 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-01-24 11:58:21 +03:00
Evgenii Stratonikov
951a7ee1c7 [#1605] policer: Do not mutate slice under iteration 
Nothing wrong with it, besides being difficult to read.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-21 05:34:54 +00:00
Evgenii Stratonikov
0bcbeb26b2 [#1605] policer: Simplify processRepNodes() checks 
Current flow is hard to reason about, #1601 is a notorious example of
accidental complexity.
1. Remove multiple nested ifs, use depth=1.
2. Process each status exactly once, hopefully preventing bugs like
   #1601.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-21 05:34:54 +00:00
Evgenii Stratonikov
c98357606b
[#1606] Use slices.Clone()/bytes.Clone() where possible 
gopatch:
```
@@
var from, to expression
@@
+import "bytes"
-to := make([]byte, len(from))
-copy(to, from)
+to := bytes.Clone(from)

@@
var from, to expression
@@
+import "bytes"
-to = make([]byte, len(from))
-copy(to, from)
+to = bytes.Clone(from)

@@
var from, to, typ expression
@@
+import "slices"
-to := make([]typ, len(from))
-copy(to, from)
+to := slices.Clone(from)

@@
var from, to, typ expression
@@
+import "slices"
-to = make([]typ, len(from))
-copy(to, from)
+to = slices.Clone(from)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-17 14:50:14 +03:00
Alexander Chuprov
80de5d70bf [#1593] node: Fix initialization of ape_chain cache 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2025-01-17 08:58:47 +00:00
Evgenii Stratonikov
57efa0bc8e
[#1604] policer: Properly handle maintenance nodes 
Consider `REP 1 REP 1` placement (selects/filters are omitted).
The placement is `[1, 2], [1, 0]`. We are the 0-th node.
Node 1 is under maintenance, so we do not replicate object
on the node 2. In the second replication group node 1 is under maintenance,
but current caching logic considers it as "replica holder" and removes
local copy. Voilà, we have DL if the object is missing from the node 1.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-16 16:37:52 +03:00
Evgenii Stratonikov
26e0c82fb8
[#1604] policer/test: Add test for MAINTENANCE runtime status 
The node can have MAINTENANCE status in the network map, but can also be
ONLINE while responding with MAINTENANCE. These are 2 different code
paths, let's test them separately.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2025-01-16 16:37:16 +03:00