a49f0717b3
[ #1685 ] metabase: Cache frequently accessed singleton buckets
...
There are some buckets we access almost always, to check whether an
object is alive. In search we also iterate over lots of objects, and
`tx.Bucket()` shows itself a lot in pprof.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ 1 │ 2 │
│ sec/op │ sec/op vs base │
Select/string_equal-8 4.753m ± 6% 3.969m ± 14% -16.50% (p=0.000 n=10)
Select/string_not_equal-8 4.247m ± 9% 3.486m ± 11% -17.93% (p=0.000 n=10)
Select/common_prefix-8 4.163m ± 5% 3.323m ± 5% -20.18% (p=0.000 n=10)
Select/unknown-8 3.557m ± 3% 3.064m ± 8% -13.85% (p=0.001 n=10)
geomean 4.158m 3.445m -17.15%
│ 1 │ 2 │
│ B/op │ B/op vs base │
Select/string_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/string_not_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/common_prefix-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/unknown-8 2.243Mi ± 0% 1.900Mi ± 0% -15.29% (p=0.000 n=10)
geomean 2.248Mi 1.905Mi -15.26%
│ 1 │ 2 │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/string_not_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/common_prefix-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/unknown-8 55.03k ± 0% 46.04k ± 0% -16.34% (p=0.000 n=10)
geomean 55.78k 46.78k -16.12%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 10:17:42 +00:00
a7ac30da9c
[ #1642 ] tree: Refactor getSortedSubTree
...
* Reuse `item` as result for `forest.TreeSortedByFilename`
invocation.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
39f549a7ab
[ #1642 ] tree: Intoduce a helper LastChild
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
760b6a44ea
[ #1642 ] tree: Fix sorted getSubtree for multiversion filenames
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a11b2d27e4
[ #1642 ] tree: Introduce Cursor type
...
* Use `Cursor` as parameter for `TreeSortedByFilename`
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a405fb1f39
[ #1683 ] metabase: Check object status once in Select()
...
objectStatus() is called twice for the same object:
First, in selectObject() to filter removed objects.
Then, again, in getObjectForSlowFilters() via db.get().
The second call will return the same result, so remove useless branch.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ status │
│ sec/op │ sec/op vs base │
Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10)
Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10)
Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10)
Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10)
geomean 5.045m 3.865m -23.39%
│ old │ status │
│ B/op │ B/op vs base │
Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10)
geomean 2.683Mi 2.248Mi -16.21%
│ old │ status │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10)
geomean 68.78k 55.77k -18.90%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
a7319bc979
[ #1683 ] metabase/test: Report allocs in benchmarkSelect()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
07a660fbc4
[ #1677 ] writecache: Add QoS limiter usage
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 16:23:33 +03:00
7893d763d1
[ #1673 ] logger: Add sampling for journald logger
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:03 +03:00
ff4e9b6ae1
[ #1673 ] logger: Drop unused fields
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:02 +03:00
997759994a
[ #1676 ] golangci: Enable gci linter
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-13 12:04:01 +03:00
ecb6b0793c
[ #1671 ] Use slices.ContainsFunc() where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
460e5cbccf
[ #1671 ] Use slices.Delete() where possible
...
gopatch is missing for this one, because
https://github.com/uber-go/gopatch/issues/179
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
155d3ddb6e
[ #1671 ] Use min builtin where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
40536d8a06
[ #1671 ] Use fmt.Appendf where warranted
...
Fix gopls warnings:
```
cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf
````
gopatch:
```
@@
var f expression
@@
-[]byte(fmt.Sprintf(f, ...))
+fmt.Appendf(nil, f, ...)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
bcc84c85a0
[ #1671 ] Replace interface{} with any
...
gopatch:
```
@@
@@
-interface{}
+any
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
2005fdda09
[ #1667 ] shard: Drop shard pool
...
After adding an ops limiter, shard's `put` pool is redundant.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 13:59:51 +03:00
597bce7a87
[ #1653 ] treeSvc: Add operations by IO tag metric
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
4ed2bbdb0f
[ #1653 ] objectSvc: Add operations by IO tag metric
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
3727d60331
[ #1653 ] qos: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
8643e0abc5
[ #1668 ] writecache: Use object size to check free space
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-10 17:52:57 +03:00
bd61f7bf0a
[ #1666 ] audit: Fix duplicated log in Patch method
...
When we do `object patch` with audit enabled we get several
duplicated entries in logs.
`object patch` request is logged in 2 places:
1. `(*auditPatchStream) CloseAndRecv()` - when the client closes
the request stream or when stream gets aborted.
2. `(*auditPatchStream) Send()` - when stream was NOT aborted.
`Send()` doesn't check if `err != nil` before logging.
It led to to logging on every `Send()` call.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 13:27:07 +00:00
9e31cb249f
[ #1635 ] control: Add method to search shards by object
...
Added method `ListShardsForObject` to ControlService and to
StorageEngine. It returns information about shards storing
object on the node.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
6260d703ce
[ #1665 ] treesvc: Disable service config query
...
By default, gRPC fetches TXT report while resolving a domain.
0914bba6c5/internal/resolver/dns/dns_resolver.go (L336)TrueCloudLab/frostfs-sdk-go#342
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-06 15:15:31 +03:00
471aeeaff3
[ #1659 ] audit: Fix duplicated request logs
...
When we do `object put` with audit enabled we get several entries
in logs: with and without object id.
`object put` request is logged in 2 places:
1. `(*auditPutStream) CloseAndRecv()` - when the client closes the
request stream or when stream gets aborted.
2. `(*auditPutStream) Send()` - when stream was NOT aborted.
`Send()` does error check for `ErrAbortStream` because if there
is any other error - CloseAndRecv will not be called and there
won't be any audit log about failed request.
It led to logging on every object chunck put, even if `err == nil`.
Added check for `err != nil` in `Send()` to fix it.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-06 12:36:16 +03:00
4c8f9580a1
[ #1662 ] object: Fix CloseAndRecv for patch streamer
...
* A client may open stream to server, not send anything and close
the open stream immediatly. This shouldn't cause a panic;
* Return the error if `s.patcher` is uninitialized. Uninitialized
patcher cannot be closed, this causes a panic.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-05 17:16:10 +03:00
4685afb1dc
[ #1636 ] engine: Validate limiter release in unit tests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:30 +03:00
eb8b9b2b3b
[ #1636 ] blobovniczatree: Validate limiter release in rebuild unit tests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:30 +03:00
6c6e463b73
[ #1636 ] shard: Change ops limiter on shard reload
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:29 +03:00
c2d855aedd
[ #1636 ] qos: Return Resource Exhausted error
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:28 +03:00
2162f8e189
[ #1636 ] object: Fix IO tag adjustment for Put/Patch
...
There was no tag adjustment for CloseAndRecv.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:28 +03:00
b9360be1dc
[ #1636 ] blobovniczatree: Use RebuildLimiter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:27 +03:00
ceff5e1f6a
[ #1636 ] storage: Refactor shard rebuild
...
Drop redundant interfaces.
Rename fields.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:27 +03:00
e0dc3c3d0c
[ #1636 ] shard: Add limiter usage
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:26 +03:00
0991077cb3
[ #1657 ] engine: Fix data race in evacuation tests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 14:14:12 +00:00
dce269c62e
[ #1639 ] services/object: Remove limiting pools for Put operation
...
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-28 11:08:09 +00:00
9a0507704a
[ #1651 ] apemanager: Wrap some errors with InvalidArgument status
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-02-27 18:25:09 +00:00
003d568ae2
[ #1628 ] innerring: Relax container homomorphic hashing check
...
Our initial desire was to prohibit using homomorphic hashing on the
network level because of the resource consumption. However, the ability
to use it, doesn't mean that we must. So only fail validation if
container wants to have homomorphic hashing, but the network prohibits
it.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-02-27 10:20:08 +03:00
02f3a7f65c
[ #1648 ] writecache: Fix race condition when reporting cache size metrics
...
There is a race condition when multiple cache operation try to report
the cache size metrics simultaneously. Consider the following example:
- the initial total size of objects stored in the cache size is 2
- worker X deletes an object and reads the cache size, which is 1
- worker Y deletes an object and reads the cache size, which is 0
- worker Y reports the cache size it learnt, which is 0
- worker X reports the cache size it learnt, which is 1
As a result, the observed cache size is 1 (i. e. one object remains
in the cache), which is incorrect because the actual cache size is 0.
To fix this, let's report the metrics periodically in the flush loop.
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-02-19 17:05:40 +03:00
5d79abe523
[ #1608 ] shard: Add IO tag for rebuild
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:07 +03:00
0671c277db
[ #1608 ] tree: Add IO tag for tree sync requests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:06 +03:00
92450a76ba
[ #1608 ] shard: Add IO tag to GC
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:06 +03:00
abba5b2089
[ #1608 ] writecache: Add IO tag to flush worker
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:06 +03:00
bfe325e61d
[ #1608 ] policer: Add IO tag to context
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:05 +03:00
acec938b2d
[ #1608 ] qos: Add client grpc interceptors
...
`qos` client interceptors replace internal IO tags `writecache`,
`policer` and `background` with `internal` IO tag for outcomming RPC.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:04 +03:00
170860c14a
[ #1608 ] logger: Add IO tag logging
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:03 +03:00
12da2f8262
[ #1608 ] object: Add IO tag adjustment layer
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:03 +03:00
f6b3f79e89
[ #1608 ] qos: Add qos service to adjust incoming IO tags
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:43:03 +03:00
9729f31e5c
[ #1608 ] grpc: Add QoS interceptors for server and clients
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-07 15:41:06 +03:00
9b113c3156
[ #1613 ] morph: Add tracing for morph queries to neo-go
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-02-05 16:38:20 +03:00
