baa2b133a2
[ #1695 ] qos: Sort tags by asc
...
Change-Id: Ia23e392bb49d2536096de2ba07fc6f8fb7ac0489
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:03 +03:00
5385f9994f
[ #1695 ] mod: Bump frostfs-observability version
...
Change-Id: Id362b71f743ff70c8cd374030c9fa67e2566022f
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:02 +03:00
eea46a599d
[ #1695 ] qos: Add treesync tag
...
Tree sync is too much different from GC and rebuild to use the same tag for GC and tree sync.
Change-Id: Ib44d5fa9a88daff507d759d0b0410cc9272e236f
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-21 13:28:02 +03:00
049a650b89
[ #1619 ] logger: Simplify logger config reloading
...
Change-Id: Ide892b250304b8cdb6c279f5f728c3b35f05df54
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2025-03-21 09:03:54 +00:00
3f4717a37f
[ #1692 ] metabase: Do not allocate map in cache unless needed
...
Change-Id: I8b1015a8c7c3df4153a08fdb788117d9f0d6c333
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 08:56:32 +00:00
60cea8c714
[ #1692 ] metabase/test: Fix end of iteration error check
...
This is not good:
```
BenchmarkListWithCursor/1_item-8 --- FAIL: BenchmarkListWithCursor/1_item-8
list_test.go:63: error: end of object listing
```
Change-Id: I61b70937ce30fefaf16ebeb0cdb51bdd39096061
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-21 08:56:18 +00:00
7df2912a83
[ #1689 ] Makefile: Create prepare-commit-msg hook too
...
`commit-msg` is ignored when `--no-verify` option is used, so there is
no way to ignore `pre-commit` while retaining `commit-msg` hook.
Ignoring pre-commit is useful, though, so we might add Change-Id in
`prepare-commit-msg` hook instead. It accepts more parameters, but the
first one is a file with the commit message, so we may reuse
`commit-msg` hook.
Change-Id: I4edb79810bbe38a5dcf7f4f07535f34c6bda0da3
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 18:01:53 +03:00
affab25512
Makefile: Add Gerrit-related targets
...
This commit adds helper targets to easily setup an existing repo for
work with Gerrit.
Change-Id: I0696eb8ea84cc16a9482be6a2fb0382fe624bb96
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 16:48:04 +03:00
45b7796151
[ #1689 ] ci: Reimplement CI tasks in Jenkinsfile
...
This commit introduces Jenkins pipeline that duplicates the features of
existing Forgejo Actions workflows.
Change-Id: I657a6c27373a1ed4736ae27b4fb660e0ac86012d
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2025-03-20 12:55:30 +00:00
e8801dbf49
[ #1691 ] metabase: Move cheaper conditions to the front in ListWithCursor()
...
`objectLocked` call is expensive, it does IO. We may omit it if object
is not expired.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:36 +00:00
eb9df85b98
[ #1685 ] metabase: Cache primary bucket
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ expired │ primary │
│ sec/op │ sec/op vs base │
Select/string_equal-8 3.529m ± 11% 3.689m ± 7% +4.55% (p=0.023 n=10)
Select/string_not_equal-8 3.440m ± 7% 3.543m ± 13% ~ (p=0.190 n=10)
Select/common_prefix-8 3.240m ± 6% 3.050m ± 5% -5.85% (p=0.005 n=10)
Select/unknown-8 3.198m ± 6% 2.928m ± 8% -8.44% (p=0.003 n=10)
geomean 3.349m 3.287m -1.84%
│ expired │ primary │
│ B/op │ B/op vs base │
Select/string_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/string_not_equal-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/common_prefix-8 1.885Mi ± 0% 1.786Mi ± 0% -5.23% (p=0.000 n=10)
Select/unknown-8 1.877Mi ± 0% 1.779Mi ± 0% -5.26% (p=0.000 n=10)
geomean 1.883Mi 1.784Mi -5.24%
│ expired │ primary │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/string_not_equal-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/common_prefix-8 46.04k ± 0% 43.04k ± 0% -6.50% (p=0.000 n=10)
Select/unknown-8 45.05k ± 0% 42.05k ± 0% -6.65% (p=0.000 n=10)
geomean 45.79k 42.79k -6.54%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:01 +00:00
21bed3362c
[ #1685 ] metabase: Cache expired bucket
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ master │ expired │
│ sec/op │ sec/op vs base │
Select/string_equal-8 4.007m ± 10% 3.529m ± 11% -11.94% (p=0.000 n=10)
Select/string_not_equal-8 3.834m ± 12% 3.440m ± 7% -10.29% (p=0.029 n=10)
Select/common_prefix-8 3.470m ± 9% 3.240m ± 6% ~ (p=0.105 n=10)
Select/unknown-8 3.156m ± 3% 3.198m ± 6% ~ (p=0.631 n=10)
geomean 3.602m 3.349m -7.03%
│ master │ expired │
│ B/op │ B/op vs base │
Select/string_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/string_not_equal-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/common_prefix-8 1.907Mi ± 0% 1.885Mi ± 0% -1.18% (p=0.000 n=10)
Select/unknown-8 1.900Mi ± 0% 1.877Mi ± 0% -1.18% (p=0.000 n=10)
geomean 1.905Mi 1.883Mi -1.18%
│ master │ expired │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/string_not_equal-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/common_prefix-8 47.03k ± 0% 46.04k ± 0% -2.12% (p=0.000 n=10)
Select/unknown-8 46.04k ± 0% 45.05k ± 0% -2.16% (p=0.000 n=10)
geomean 46.78k 45.79k -2.13%
```
Change-Id: I9c7a5e1f5c8b9eb3f25a563fd74c6ad2a9d1b92e
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 12:52:01 +00:00
af5b3575d0
[ #1690 ] qos: Do not export zero metrics counters
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-20 14:42:35 +03:00
a49f0717b3
[ #1685 ] metabase: Cache frequently accessed singleton buckets
...
There are some buckets we access almost always, to check whether an
object is alive. In search we also iterate over lots of objects, and
`tx.Bucket()` shows itself a lot in pprof.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ 1 │ 2 │
│ sec/op │ sec/op vs base │
Select/string_equal-8 4.753m ± 6% 3.969m ± 14% -16.50% (p=0.000 n=10)
Select/string_not_equal-8 4.247m ± 9% 3.486m ± 11% -17.93% (p=0.000 n=10)
Select/common_prefix-8 4.163m ± 5% 3.323m ± 5% -20.18% (p=0.000 n=10)
Select/unknown-8 3.557m ± 3% 3.064m ± 8% -13.85% (p=0.001 n=10)
geomean 4.158m 3.445m -17.15%
│ 1 │ 2 │
│ B/op │ B/op vs base │
Select/string_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/string_not_equal-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/common_prefix-8 2.250Mi ± 0% 1.907Mi ± 0% -15.24% (p=0.000 n=10)
Select/unknown-8 2.243Mi ± 0% 1.900Mi ± 0% -15.29% (p=0.000 n=10)
geomean 2.248Mi 1.905Mi -15.26%
│ 1 │ 2 │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/string_not_equal-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/common_prefix-8 56.02k ± 0% 47.03k ± 0% -16.05% (p=0.000 n=10)
Select/unknown-8 55.03k ± 0% 46.04k ± 0% -16.34% (p=0.000 n=10)
geomean 55.78k 46.78k -16.12%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-20 10:17:42 +00:00
a7ac30da9c
[ #1642 ] tree: Refactor getSortedSubTree
...
* Reuse `item` as result for `forest.TreeSortedByFilename`
invocation.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
39f549a7ab
[ #1642 ] tree: Intoduce a helper LastChild
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
760b6a44ea
[ #1642 ] tree: Fix sorted getSubtree for multiversion filenames
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a11b2d27e4
[ #1642 ] tree: Introduce Cursor type
...
* Use `Cursor` as parameter for `TreeSortedByFilename`
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-20 10:12:49 +00:00
a405fb1f39
[ #1683 ] metabase: Check object status once in Select()
...
objectStatus() is called twice for the same object:
First, in selectObject() to filter removed objects.
Then, again, in getObjectForSlowFilters() via db.get().
The second call will return the same result, so remove useless branch.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ status │
│ sec/op │ sec/op vs base │
Select/string_equal-8 5.022m ± 7% 3.968m ± 8% -20.98% (p=0.000 n=10)
Select/string_not_equal-8 4.953m ± 9% 3.990m ± 10% -19.44% (p=0.000 n=10)
Select/common_prefix-8 4.962m ± 8% 3.971m ± 9% -19.98% (p=0.000 n=10)
Select/unknown-8 5.246m ± 9% 3.548m ± 5% -32.37% (p=0.000 n=10)
geomean 5.045m 3.865m -23.39%
│ old │ status │
│ B/op │ B/op vs base │
Select/string_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/string_not_equal-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/common_prefix-8 2.685Mi ± 0% 2.250Mi ± 0% -16.20% (p=0.000 n=10)
Select/unknown-8 2.677Mi ± 0% 2.243Mi ± 0% -16.24% (p=0.000 n=10)
geomean 2.683Mi 2.248Mi -16.21%
│ old │ status │
│ allocs/op │ allocs/op vs base │
Select/string_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/string_not_equal-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/common_prefix-8 69.03k ± 0% 56.02k ± 0% -18.84% (p=0.000 n=10)
Select/unknown-8 68.03k ± 0% 55.03k ± 0% -19.11% (p=0.000 n=10)
geomean 68.78k 55.77k -18.90%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
a7319bc979
[ #1683 ] metabase/test: Report allocs in benchmarkSelect()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-18 11:48:51 +00:00
fc743cc537
[ #1684 ] cli: Correct description of control shards writecache seal
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-18 10:56:30 +03:00
54ef71a92f
[ #1680 ] Update staticcheck to 2025.1.1
...
Change-Id: Ie851e714afebf171c4d42d4c49b42379c2665113
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
91c7b39232
[ #1680 ] go.mod: Bump go version to 1.23
...
Change-Id: I77f908924f675e676f0db6a57204d7c1e0df219a
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
ef6ac751df
[ #1671 ] Update gopls to v0.17.1
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-17 10:02:53 +00:00
fde2649e60
[ #1678 ] adm: Fix frostfs-adm morph list-subjects & list-group-subjects
...
`include-names` for `list-subjects` returns error `invalid response subject struct`
because `ListSubjects` returns only subject addresses (see frostfs-contract).
Replace `include-names` with `extended` as now all subject info printed.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 13:42:29 +00:00
07a660fbc4
[ #1677 ] writecache: Add QoS limiter usage
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-14 16:23:33 +03:00
7893d763d1
[ #1673 ] logger: Add sampling for journald logger
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:03 +03:00
ff4e9b6ae1
[ #1673 ] logger: Drop unused fields
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-13 14:52:02 +03:00
997759994a
[ #1676 ] golangci: Enable gci linter
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2025-03-13 12:04:01 +03:00
ecb6b0793c
[ #1671 ] Use slices.ContainsFunc() where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
460e5cbccf
[ #1671 ] Use slices.Delete() where possible
...
gopatch is missing for this one, because
https://github.com/uber-go/gopatch/issues/179
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
155d3ddb6e
[ #1671 ] Use min builtin where possible
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
40536d8a06
[ #1671 ] Use fmt.Appendf where warranted
...
Fix gopls warnings:
```
cmd/frostfs-adm/internal/modules/morph/config/config.go:68:20-64: Replace []byte(fmt.Sprintf...) with fmt.Appendf
````
gopatch:
```
@@
var f expression
@@
-[]byte(fmt.Sprintf(f, ...))
+fmt.Appendf(nil, f, ...)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
d66bffb191
[ #1671 ] Use max builtin where possible
...
gopatcH:
```
@@
var d, a expression
@@
-if d < a {
- d = a
-}
-return d
+return max(d, a)
@@
var d, a expression
@@
-if d <= a {
- d = a
-}
-return d
+return max(d, a)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
bcc84c85a0
[ #1671 ] Replace interface{} with any
...
gopatch:
```
@@
@@
-interface{}
+any
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-13 08:12:20 +00:00
737788b35f
[ #1669 ] go.mod: Bump frostfs-qos version
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-12 09:23:53 +03:00
2005fdda09
[ #1667 ] shard: Drop shard pool
...
After adding an ops limiter, shard's `put` pool is redundant.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 13:59:51 +03:00
597bce7a87
[ #1653 ] treeSvc: Add operations by IO tag metric
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
4ed2bbdb0f
[ #1653 ] objectSvc: Add operations by IO tag metric
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
3727d60331
[ #1653 ] qos: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
d36afa31c7
[ #1653 ] qos: Fix logging
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-11 10:57:47 +00:00
8643e0abc5
[ #1668 ] writecache: Use object size to check free space
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2025-03-10 17:52:57 +03:00
bd61f7bf0a
[ #1666 ] audit: Fix duplicated log in Patch method
...
When we do `object patch` with audit enabled we get several
duplicated entries in logs.
`object patch` request is logged in 2 places:
1. `(*auditPatchStream) CloseAndRecv()` - when the client closes
the request stream or when stream gets aborted.
2. `(*auditPatchStream) Send()` - when stream was NOT aborted.
`Send()` doesn't check if `err != nil` before logging.
It led to to logging on every `Send()` call.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 13:27:07 +00:00
df6d9da82a
[ #1635 ] cli: Add command to get object's shard info
...
Added `frostfs-cli object locate` subcommand. It lists info
about shards storing an object.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
aab8addae0
[ #1635 ] cli: Make object.readObjectAddress() public
...
This method will be useful for upcoming control command.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
9e31cb249f
[ #1635 ] control: Add method to search shards by object
...
Added method `ListShardsForObject` to ControlService and to
StorageEngine. It returns information about shards storing
object on the node.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-07 14:32:01 +03:00
6260d703ce
[ #1665 ] treesvc: Disable service config query
...
By default, gRPC fetches TXT report while resolving a domain.
0914bba6c5/internal/resolver/dns/dns_resolver.go (L336)TrueCloudLab/frostfs-sdk-go#342
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-06 15:15:31 +03:00
a17c3356fa
[ #1665 ] go.mod: Update sdk-go
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2025-03-06 15:15:31 +03:00
471aeeaff3
[ #1659 ] audit: Fix duplicated request logs
...
When we do `object put` with audit enabled we get several entries
in logs: with and without object id.
`object put` request is logged in 2 places:
1. `(*auditPutStream) CloseAndRecv()` - when the client closes the
request stream or when stream gets aborted.
2. `(*auditPutStream) Send()` - when stream was NOT aborted.
`Send()` does error check for `ErrAbortStream` because if there
is any other error - CloseAndRecv will not be called and there
won't be any audit log about failed request.
It led to logging on every object chunck put, even if `err == nil`.
Added check for `err != nil` in `Send()` to fix it.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2025-03-06 12:36:16 +03:00
4c8f9580a1
[ #1662 ] object: Fix CloseAndRecv for patch streamer
...
* A client may open stream to server, not send anything and close
the open stream immediatly. This shouldn't cause a panic;
* Return the error if `s.patcher` is uninitialized. Uninitialized
patcher cannot be closed, this causes a panic.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2025-03-05 17:16:10 +03:00
