8cbf9a0b44
[ #957 ] treesvc: Use sorting when number of items is small
...
DCO action / DCO (pull_request) Successful in 2m13s
Build / Build Components (1.20) (pull_request) Successful in 2m36s
Vulncheck / Vulncheck (pull_request) Successful in 3m1s
Build / Build Components (1.21) (pull_request) Successful in 4m7s
Tests and linters / Staticcheck (pull_request) Successful in 4m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m13s
Tests and linters / Lint (pull_request) Successful in 7m12s
Tests and linters / Tests with -race (pull_request) Successful in 7m27s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m30s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-06 14:14:50 +03:00
d18eaf060d
[ #957 ] treesvc: Implement pairing heap
...
With a large number of objects in a flat tree, sorting the nodes
dominates the latency of the operation. This scales nonlinearly
as O(n log n). Pairing heap has O(1) insert, and O(log n) extractMin,
which allows us to have O(n + log n) = O(n) latency on the first
operation, albeit with a slight increase in total running time.
On a real cluster with 2m objects, the latency decreased from 25s to
15s.
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ cache │ noparent │
│ sec/op │ sec/op vs base │
GetSubTree/latency-8 2349.9µ ± 19% 572.1µ ± 17% -75.65% (p=0.000 n=10)
GetSubTree/total_time-8 70.62m ± 8% 86.02m ± 3% +21.81% (p=0.000 n=10)
geomean 12.88m 7.015m -45.54%
│ cache │ noparent │
│ B/op │ B/op vs base │
GetSubTree/latency-8 43.87Mi ± 0% 32.81Mi ± 0% -25.22% (p=0.000 n=10)
GetSubTree/total_time-8 43.87Mi ± 0% 32.81Mi ± 0% -25.22% (p=0.000 n=10)
geomean 43.87Mi 32.81Mi -25.22%
│ cache │ noparent │
│ allocs/op │ allocs/op vs base │
GetSubTree/latency-8 400.0k ± 0% 500.0k ± 0% +24.99% (p=0.000 n=10)
GetSubTree/total_time-8 400.0k ± 0% 500.0k ± 0% +24.99% (p=0.000 n=10)
geomean 400.0k 500.0k +24.99%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-06 14:14:49 +03:00
77bdadde14
[ #957 ] treesvc/heap: Cache FileName attribute
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ new │ cache │
│ sec/op │ sec/op vs base │
GetSubTree/latency-8 4.059m ± 20% 2.350m ± 19% -42.11% (p=0.000 n=10)
GetSubTree/total_time-8 78.82m ± 4% 70.62m ± 8% -10.39% (p=0.001 n=10)
geomean 17.89m 12.88m -27.98%
│ new │ cache │
│ B/op │ B/op vs base │
GetSubTree/latency-8 32.81Mi ± 0% 43.87Mi ± 0% +33.72% (p=0.000 n=10)
GetSubTree/total_time-8 32.81Mi ± 0% 43.87Mi ± 0% +33.73% (p=0.000 n=10)
geomean 32.81Mi 43.87Mi +33.72%
│ new │ cache │
│ allocs/op │ allocs/op vs base │
GetSubTree/latency-8 400.0k ± 0% 400.0k ± 0% +0.01% (p=0.000 n=10)
GetSubTree/total_time-8 400.0k ± 0% 400.0k ± 0% +0.01% (p=0.000 n=10)
geomean 400.0k 400.0k +0.01%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-06 10:15:28 +03:00
3601feab81
[ #957 ] treesvc: Abstract out Heap interface for listing
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
│ old │ new │
│ sec/op │ sec/op vs base │
GetSubTree/latency-8 3.542m ± 12% 4.059m ± 20% +14.59% (p=0.005 n=10)
GetSubTree/total_time-8 73.19m ± 9% 78.82m ± 4% +7.69% (p=0.043 n=10)
geomean 16.10m 17.89m +11.09%
│ old │ new │
│ B/op │ B/op vs base │
GetSubTree/latency-8 28.23Mi ± 0% 32.81Mi ± 0% +16.22% (p=0.000 n=10)
GetSubTree/total_time-8 28.23Mi ± 0% 32.81Mi ± 0% +16.22% (p=0.000 n=10)
geomean 28.23Mi 32.81Mi +16.22%
│ old │ new │
│ allocs/op │ allocs/op vs base │
GetSubTree/latency-8 400.0k ± 0% 400.0k ± 0% +0.00% (p=0.000 n=10)
GetSubTree/total_time-8 400.0k ± 0% 400.0k ± 0% +0.00% (p=0.000 n=10)
geomean 400.0k 400.0k +0.00%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-06 10:14:35 +03:00
7eab9fcede
[ #957 ] treesvc: Add benchmark for getSubTree
...
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkGetSubTree/latency-8 16 3074441 ns/op 29602255 B/op 400010 allocs/op
BenchmarkGetSubTree/latency-8 15 3318523 ns/op 29602739 B/op 400009 allocs/op
BenchmarkGetSubTree/latency-8 16 3195982 ns/op 29602004 B/op 400009 allocs/op
BenchmarkGetSubTree/latency-8 15 3584962 ns/op 29603457 B/op 400009 allocs/op
BenchmarkGetSubTree/latency-8 15 3623496 ns/op 29602061 B/op 400009 allocs/op
BenchmarkGetSubTree/latency-8 15 3346428 ns/op 29602054 B/op 400009 allocs/op
BenchmarkGetSubTree/latency-8 16 3965596 ns/op 29602029 B/op 400009 allocs/op
BenchmarkGetSubTree/latency-8 12 4439678 ns/op 29604445 B/op 400009 allocs/op
BenchmarkGetSubTree/latency-8 15 3499192 ns/op 29602053 B/op 400009 allocs/op
BenchmarkGetSubTree/latency-8 14 3896512 ns/op 29601542 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 15 70107430 ns/op 29602047 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 13 79811187 ns/op 29601585 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 15 73591921 ns/op 29602068 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 15 77711739 ns/op 29602047 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 15 71107422 ns/op 29602140 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 15 80340752 ns/op 29601606 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 16 72185626 ns/op 29607127 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 16 79709510 ns/op 29601985 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 16 72779459 ns/op 29601474 B/op 400009 allocs/op
BenchmarkGetSubTree/total_time-8 15 69887160 ns/op 29600968 B/op 400009 allocs/op
PASS
ok git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree 25.032s
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-03 00:19:22 +03:00
95e15f499f
[ #922 ] Update files generated by protoc
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-02 12:09:51 +00:00
f1b2b8bffa
[ #895 ] test: Fix NewLogger arguments list
...
`debug` is always true.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
c19396d203
[ #885 ] control: Make chain id bytes in grpc
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-01-11 07:24:22 +00:00
11add38e87
[ #857 ] golangci: Add protogetter linter
...
DCO action / DCO (pull_request) Successful in 1m38s
Vulncheck / Vulncheck (pull_request) Successful in 3m2s
Build / Build Components (1.21) (pull_request) Successful in 3m51s
Build / Build Components (1.20) (pull_request) Successful in 4m3s
Tests and linters / Staticcheck (pull_request) Successful in 5m17s
Tests and linters / Lint (pull_request) Successful in 6m11s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m7s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m32s
Tests and linters / Tests with -race (pull_request) Successful in 12m22s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
26e4f7005c
[ #741 ] treesvc: Refactor tree sync
...
Fix linter issues.
Add error logging.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
b21be1abdd
[ #741 ] treesvc: Do not update sync height if some node is unavailable
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
b215817e14
[ #741 ] treesvc: Remove unused height variables
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
7cdae4f660
[ #792 ] proto: Regenerate with fixed version
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:09:13 +00:00
79088baa06
[ #772 ] node: Apply gofumpt
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
c1e4130020
[ #146 ] node: Add trace_id to logs
...
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
DCO action / DCO (pull_request) Successful in 3m36s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Tests and linters / Lint (pull_request) Successful in 5m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
4ea0df77d0
[ #574 ] policer: Check if the container was really removed
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
554ff2c06b
[ #574 ] core: Extend Source interface with DeletionInfo method
...
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
96e690883f
[ #638 ] Unify test loggers
...
In some places we have debug=false, in others debug=true.
Let's be consistent.
Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
322c1dc273
[ #638 ] Use test.NewLogger() in tests
...
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"
-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
23be3eb627
[ #574 ] tree: Check if container is really removed
...
Vulncheck / Vulncheck (pull_request) Successful in 2m37s
Build / Build Components (1.21) (pull_request) Successful in 3m36s
Tests and linters / Staticcheck (pull_request) Successful in 3m36s
DCO action / DCO (pull_request) Successful in 4m16s
Tests and linters / Lint (pull_request) Successful in 5m34s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m36s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m46s
Tests and linters / Tests with -race (pull_request) Successful in 6m21s
Build / Build Components (1.20) (pull_request) Successful in 8m1s
* Use DeletionInfo method from morph client to check if
the container has been really removed from neo-go
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-21 12:50:20 +03:00
f7042c5a6f
[ #609 ] Replace zaptest.NewLogger() with zap.L()
...
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"
-zaptest.NewLogger(t)
+zap.L()
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:51:10 +00:00
e604a3d749
[ #607 ] *: Use zap.Stringer() where possible
...
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
f(
...,
- zap.String(t, a.String()),
+ zap.Stringer(t, a),
...,
)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
1e8b4b8a17
[ #557 ] services: Regenerate stable marshalers
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-02 13:37:40 +00:00
eed594431f
[ #335 ] treesvc: Add GetSubTree ordering unit test
...
Build / Build Components (1.20) (pull_request) Successful in 3m47s
Build / Build Components (1.19) (pull_request) Successful in 4m2s
Tests and linters / Tests (1.20) (pull_request) Successful in 3m8s
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Staticcheck (pull_request) Successful in 3m48s
Tests and linters / Tests with -race (pull_request) Successful in 7m59s
Tests and linters / Tests (1.19) (pull_request) Successful in 15m36s
Tests and linters / Lint (pull_request) Successful in 18m12s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
b4e72a2dfd
[ #335 ] treesvc: Sort nodes by Filename in GetSubTree
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
8a9fc2c372
[ #510 ] treesvc: Rename tableFromBearer to useBearer
...
ci/woodpecker/push/pre-commit Pipeline was successful
With impersonation, the old name is no longer descriptive.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
b8bcfac531
[ #510 ] treesvc: Fix panic in bearer token processing
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
6eefe9747e
treesvc: Do not provide credentials unless TLS is used
...
ci/woodpecker/push/pre-commit Pipeline failed
4f413fe86e
2023-07-06 08:04:28 +00:00
d0ab552a90
[ #478 ] *: Fix funlen linter warnings
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
8a4e250dae
[ #468 ] *: replace outdated TODO crypto-related links
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
4f413fe86e
[ #1 ] treesvc: Properly check for secure transport
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
71889234b7
[ #449 ] tree: Allow reading requests signed by keys from allow list
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-23 14:44:19 +03:00
4887f489a1
[ #17 ] Add morph client metrics
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 14:06:04 +03:00
90e9247b69
[ #371 ] ir: Add morph cache metrics
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-13 10:10:13 +00:00
957a43a124
[ #266 ] services/tree: Add sync check
...
ci/woodpecker/push/pre-commit Pipeline was successful
Do not accept requests until initial sync is finished.
`Apply` is deliberately left out -- we don't want to miss anything new.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
e69a1e8482
[ #266 ] services/tree: Return operation log up to some height
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
dcdfb6ed41
[ #412 ] node: Use observability interceptors
...
Use metrics and tracing interceptors.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
74578052f9
[ #412 ] node: Replace tracing package
...
Use observability module.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
bc34fee6a7
[ #370 ] Add tree service metrics
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-26 13:39:12 +00:00
13a7a90101
[ #355 ] Increase tree svc client cache size to test hypotheses
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 14:11:46 +03:00
Pavel Karpy
479c5a65e1
[ #322 ] node: Fix tree svc panic
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-04 19:48:51 +03:00
529d0bc710
[ #302 ] tree: Drop unused ctx
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 16:54:06 +03:00
ff25521204
[ #270 ] Add IR epoch tick control call
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-28 07:57:00 +00:00
f07e2d4812
[ #285 ] lint: Fix revive/unused-parameter
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
9d01029733
[ #166 ] node: Parallelize background tree service sync by batching
...
ci/woodpecker/push/pre-commit Pipeline was successful
* Merge operations
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
299b24b974
[ #166 ] node: Parallelize background tree service sync by batching
...
* Concurrently dispatch TreeApply operations for batching in forest
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
Denis Kirillov
dce5924a89
[ #229 ] services/tree: Use bearer owner as signer
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 10:23:33 +03:00
89530534a1
[ #229 ] service/tree: Disable container owner check in tree service
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
f07d4158f5
[ #249 ] node: Drop subnet from IR and morph
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
6121b541b5
[ #242 ] treesvc: Add tracing spans
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
