72f0d1e24e
[ #1692 ] metabase: Remove useless count variable
...
It is always equal to `len(to)`.
Change-Id: Id7a4c26e9711216b78f96e6b2511efa0773e3471
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 17:56:08 +03:00
80bd5efe3f
[ #1692 ] metabase: Use bucket cache in ListWithCursor()
...
No changes in speed, but unified approach:
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ master │ new │
│ sec/op │ sec/op vs base │
ListWithCursor/1_item-8 6.067µ ± 8% 5.731µ ± 10% ~ (p=0.052 n=10)
ListWithCursor/10_items-8 25.40µ ± 11% 26.12µ ± 13% ~ (p=0.971 n=10)
ListWithCursor/100_items-8 210.7µ ± 9% 203.2µ ± 6% ~ (p=0.280 n=10)
geomean 31.90µ 31.22µ -2.16%
│ master │ new │
│ B/op │ B/op vs base │
ListWithCursor/1_item-8 3.287Ki ± 0% 3.287Ki ± 0% ~ (p=1.000 n=10) ¹
ListWithCursor/10_items-8 15.63Ki ± 0% 15.62Ki ± 0% ~ (p=0.328 n=10)
ListWithCursor/100_items-8 138.1Ki ± 0% 138.1Ki ± 0% ~ (p=0.340 n=10)
geomean 19.21Ki 19.21Ki -0.00%
¹ all samples are equal
│ master │ new │
│ allocs/op │ allocs/op vs base │
ListWithCursor/1_item-8 109.0 ± 0% 109.0 ± 0% ~ (p=1.000 n=10) ¹
ListWithCursor/10_items-8 380.0 ± 0% 380.0 ± 0% ~ (p=1.000 n=10) ¹
ListWithCursor/100_items-8 3.082k ± 0% 3.082k ± 0% ~ (p=1.000 n=10) ¹
geomean 503.5 503.5 +0.00%
¹ all samples are equal
```
Change-Id: Ic11673427615053656b2a60068a6d4dbd27af2cb
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 17:56:08 +03:00
1e660c6c61
[ #1692 ] metabase: Do not allocate map in cache unless needed
...
Change-Id: I8b1015a8c7c3df4153a08fdb788117d9f0d6c333
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 17:56:08 +03:00
7eb2f6d0e1
[ #1692 ] metabase/test: Fix end of iteration error check
...
This is not good:
```
BenchmarkListWithCursor/1_item-8 --- FAIL: BenchmarkListWithCursor/1_item-8
list_test.go:63: error: end of object listing
```
Change-Id: I61b70937ce30fefaf16ebeb0cdb51bdd39096061
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 17:56:08 +03:00
45b7796151
[ #1689 ] ci: Reimplement CI tasks in Jenkinsfile
...
This commit introduces Jenkins pipeline that duplicates the features of
existing Forgejo Actions workflows.
Change-Id: I657a6c27373a1ed4736ae27b4fb660e0ac86012d
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-20 12:55:30 +00:00
e8801dbf49
[ #1691 ] metabase: Move cheaper conditions to the front in ListWithCursor()
...
`objectLocked` call is expensive, it does IO. We may omit it if object
is not expired.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:36 +00:00
eb9df85b98
[ #1685 ] metabase: Cache primary bucket
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ expired │ primary │
│ sec/op │ sec/op vs base │
Select/string_equal-8 3.529m ± 11% 3.689m ± 7% +4.55% (p=0.023 n=10)
Select/string_not_equal-8 3.440m ± 7% 3.543m ± 13% ~ (p=0.190 n=10)
Select/common_prefix-8 3.240m ± 6% 3.050m ± 5% -5.85% (p=0.005 n=10)
Select/unknown-8 3.198m ± 6% 2.928m ± 8% -8.44% (p=0.003 n=10)
geomean 3.349m 3.287m -1.84%
│ expired │ primary │
│ B/op │ B/op vs base │
Select/string_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/string_not_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/common_prefix-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/unknown-8 1.877Mi ± 0% 1.779Mi ± 0% -5.26% (p=0.000 n=10)
geomean 1.883Mi 1.784Mi -5.24%
│ expired │ primary │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/string_not_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/common_prefix-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/unknown-8 45.05k ± 0% 42.05k ± 0% -6.65% (p=0.000 n=10)
geomean 45.79k 42.79k -6.54%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:01 +00:00
21bed3362c
[ #1685 ] metabase: Cache expired bucket
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ master │ expired │
│ sec/op │ sec/op vs base │
Select/string_equal-8 4.007m ± 10% 3.529m ± 11% -11.94% (p=0.000 n=10)
Select/string_not_equal-8 3.834m ± 12% 3.440m ± 7% -10.29% (p=0.029 n=10)
Select/common_prefix-8 3.470m ± 9% 3.240m ± 6% ~ (p=0.105 n=10)
Select/unknown-8 3.156m ± 3% 3.198m ± 6% ~ (p=0.631 n=10)
geomean 3.602m 3.349m -7.03%
│ master │ expired │
│ B/op │ B/op vs base │
Select/string_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/string_not_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/common_prefix-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/unknown-8 1.900Mi ± 0% 1.877Mi ± 0% -1.18% (p=0.000 n=10)
geomean 1.905Mi 1.883Mi -1.18%
│ master │ expired │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/string_not_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/common_prefix-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/unknown-8 46.04k ± 0% 45.05k ± 0% -2.16% (p=0.000 n=10)
geomean 46.78k 45.79k -2.13%
```
Change-Id: I9c7a5e1f5c8b9eb3f25a563fd74c6ad2a9d1b92e
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:01 +00:00
af5b3575d0
[ #1690 ] qos: Do not export zero metrics counters
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-20 14:42:35 +03:00
a49f0717b3
[ #1685 ] metabase: Cache frequently accessed singleton buckets
...
There are some buckets we access almost always, to check whether an
object is alive. In search we also iterate over lots of objects, and
`tx.Bucket()` shows itself a lot in pprof.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ 1 │ 2 │
│ sec/op │ sec/op vs base │
Select/string_equal-8 4.753m ± 6% 3.969m ± 14% -16.50% (p=0.000 n=10)
Select/string_not_equal-8 4.247m ± 9% 3.486m ± 11% -17.93% (p=0.000 n=10)
Select/common_prefix-8 4.163m ± 5% 3.323m ± 5% -20.18% (p=0.000 n=10)
Select/unknown-8 3.557m ± 3% 3.064m ± 8% -13.85% (p=0.001 n=10)
geomean 4.158m 3.445m -17.15%
│ 1 │ 2 │
│ B/op │ B/op vs base │
Select/string_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/string_not_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/common_prefix-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/unknown-8 2.243Mi ± 0% 1.900Mi ± 0% -15.29% (p=0.000 n=10)
geomean 2.248Mi 1.905Mi -15.26%
│ 1 │ 2 │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/string_not_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/common_prefix-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/unknown-8 55.03k ± 0% 46.04k ± 0% -16.34% (p=0.000 n=10)
geomean 55.78k 46.78k -16.12%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 10:17:42 +00:00
a7ac30da9c
[ #1642 ] tree: Refactor getSortedSubTree
...
* Reuse `item` as result for `forest.TreeSortedByFilename`
invocation.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
39f549a7ab
[ #1642 ] tree: Intoduce a helper LastChild
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
760b6a44ea
[ #1642 ] tree: Fix sorted getSubtree for multiversion filenames
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a11b2d27e4
[ #1642 ] tree: Introduce Cursor type
...
* Use `Cursor` as parameter for `TreeSortedByFilename`
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a405fb1f39
[ #1683 ] metabase: Check object status once in Select()
...
objectStatus() is called twice for the same object:
First, in selectObject() to filter removed objects.
Then, again, in getObjectForSlowFilters() via db.get().
The second call will return the same result, so remove useless branch.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ status │
│ sec/op │ sec/op vs base │
Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10)
Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10)
Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10)
Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10)
geomean 5.045m 3.865m -23.39%
│ old │ status │
│ B/op │ B/op vs base │
Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10)
geomean 2.683Mi 2.248Mi -16.21%
│ old │ status │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10)
geomean 68.78k 55.77k -18.90%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
a7319bc979
[ #1683 ] metabase/test: Report allocs in benchmarkSelect()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
fc743cc537
[ #1684 ] cli: Correct description of control shards writecache seal
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-18 10:56:30 +03:00
54ef71a92f
[ #1680 ] Update staticcheck to 2025.1.1
...
Change-Id: Ie851e714afebf171c4d42d4c49b42379c2665113
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
91c7b39232
[ #1680 ] go.mod: Bump go version to 1.23
...
Change-Id: I77f908924f675e676f0db6a57204d7c1e0df219a
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
ef6ac751df
[ #1671 ] Update gopls to v0.17.1
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
fde2649e60
[ #1678 ] adm: Fix frostfs-adm morph list-subjects & list-group-subjects
...
`include-names` for `list-subjects` returns error `invalid response subject struct`
because `ListSubjects` returns only subject addresses (see frostfs-contract).
Replace `include-names` with `extended` as now all subject info printed.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 13:42:29 +00:00
07a660fbc4
[ #1677 ] writecache: Add QoS limiter usage
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 16:23:33 +03:00
7893d763d1
[ #1673 ] logger: Add sampling for journald logger
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:03 +03:00
ff4e9b6ae1
[ #1673 ] logger: Drop unused fields
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:02 +03:00
997759994a
[ #1676 ] golangci: Enable gci linter
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-13 12:04:01 +03:00
ecb6b0793c
[ #1671 ] Use slices.ContainsFunc() where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
460e5cbccf
[ #1671 ] Use slices.Delete() where possible
...
gopatch is missing for this one, because
https://github.com/uber-go/gopatch/issues/179
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
155d3ddb6e
[ #1671 ] Use min builtin where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
40536d8a06
[ #1671 ] Use fmt.Appendf where warranted
...
Fix gopls warnings:
```
cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf
````
gopatch:
```
@@
var f expression
@@
-[]byte(fmt.Sprintf(f, ...))
+fmt.Appendf(nil, f, ...)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
d66bffb191
[ #1671 ] Use max builtin where possible
...
gopatcH:
```
@@
var d, a expression
@@
-if d < a {
- d = a
-}
-return d
+return max(d, a)
@@
var d, a expression
@@
-if d <= a {
- d = a
-}
-return d
+return max(d, a)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
bcc84c85a0
[ #1671 ] Replace interface{} with any
...
gopatch:
```
@@
@@
-interface{}
+any
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
737788b35f
[ #1669 ] go.mod: Bump frostfs-qos version
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-12 09:23:53 +03:00
2005fdda09
[ #1667 ] shard: Drop shard pool
...
After adding an ops limiter, shard's `put` pool is redundant.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 13:59:51 +03:00
597bce7a87
[ #1653 ] treeSvc: Add operations by IO tag metric
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
4ed2bbdb0f
[ #1653 ] objectSvc: Add operations by IO tag metric
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
3727d60331
[ #1653 ] qos: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
d36afa31c7
[ #1653 ] qos: Fix logging
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
8643e0abc5
[ #1668 ] writecache: Use object size to check free space
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-10 17:52:57 +03:00
bd61f7bf0a
[ #1666 ] audit: Fix duplicated log in Patch method
...
When we do `object patch` with audit enabled we get several
duplicated entries in logs.
`object patch` request is logged in 2 places:
1. `(*auditPatchStream) CloseAndRecv()` - when the client closes
the request stream or when stream gets aborted.
2. `(*auditPatchStream) Send()` - when stream was NOT aborted.
`Send()` doesn't check if `err != nil` before logging.
It led to to logging on every `Send()` call.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 13:27:07 +00:00
df6d9da82a
[ #1635 ] cli: Add command to get object's shard info
...
Added `frostfs-cli object locate` subcommand. It lists info
about shards storing an object.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
aab8addae0
[ #1635 ] cli: Make object.readObjectAddress() public
...
This method will be useful for upcoming control command.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
9e31cb249f
[ #1635 ] control: Add method to search shards by object
...
Added method `ListShardsForObject` to ControlService and to
StorageEngine. It returns information about shards storing
object on the node.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
6260d703ce
[ #1665 ] treesvc: Disable service config query
...
By default, gRPC fetches TXT report while resolving a domain.
0914bba6c5/internal/resolver/dns/dns_resolver.go (L336)TrueCloudLab/frostfs-sdk-go#342
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-06 15:15:31 +03:00
a17c3356fa
[ #1665 ] go.mod: Update sdk-go
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-06 15:15:31 +03:00
471aeeaff3
[ #1659 ] audit: Fix duplicated request logs
...
When we do `object put` with audit enabled we get several entries
in logs: with and without object id.
`object put` request is logged in 2 places:
1. `(*auditPutStream) CloseAndRecv()` - when the client closes the
request stream or when stream gets aborted.
2. `(*auditPutStream) Send()` - when stream was NOT aborted.
`Send()` does error check for `ErrAbortStream` because if there
is any other error - CloseAndRecv will not be called and there
won't be any audit log about failed request.
It led to logging on every object chunck put, even if `err == nil`.
Added check for `err != nil` in `Send()` to fix it.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-06 12:36:16 +03:00
4c8f9580a1
[ #1662 ] object: Fix CloseAndRecv for patch streamer
...
* A client may open stream to server, not send anything and close
the open stream immediatly. This shouldn't cause a panic;
* Return the error if `s.patcher` is uninitialized. Uninitialized
patcher cannot be closed, this causes a panic.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-05 17:16:10 +03:00
bf8914fedc
[ #1416 ] lens/explorer: Support metabase schema v3
...
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2025-03-05 07:20:05 +00:00
5ba0e2918e
[ #1661 ] cli: Clarify --rpc-endpoint values
...
`multiaddr` is not something an average user knows. Personally, I have
never used it in CLI. On the other hand, we need to connect with TLS quite often,
so it needs to be mentioned in help.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-04 10:19:49 +03:00
4685afb1dc
[ #1636 ] engine: Validate limiter release in unit tests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:30 +03:00
eb8b9b2b3b
[ #1636 ] blobovniczatree: Validate limiter release in rebuild unit tests
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-02-28 17:25:30 +03:00
