493cafc62a
[ #355 ] Increase tree svc client cache size to test hypotheses
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 15:17:35 +03:00
3711976dfc
[ #314 ] writecache: remove objects right after they are flushed
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-16 14:16:25 +03:00
Pavel Karpy
c3f5045842
[ #314 ] wc: Do not lose small objects on disk errors
...
Do return error if an object could not been stored on WC's disk.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-15 16:27:44 +03:00
Pavel Karpy
ab65063d6d
[ #314 ] wc: Simplify background workers naming
...
Also, drop not used arg.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-15 16:27:42 +03:00
Pavel Karpy
c60029d3b0
[ #323 ] node: Fix tree svc panic
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-04 20:04:30 +03:00
Pavel Karpy
0beb7ccf5c
[ #284 ] node: Use copy_number on server side
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-26 10:57:34 +03:00
79d59e4ed2
[ #266 ] services/tree: Do not accept requests until initial sync is finished
...
ci/woodpecker/pr/pre-commit Pipeline failed
ci/woodpecker/push/pre-commit Pipeline failed
`Apply` is deliberately left out -- we don't want to miss anything new.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
364b4ac572
[ #266 ] services/tree: Batch operations on the service level
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
f7679a8168
[ #266 ] services/tree: Return operation log up to some height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
2dc2fe8780
[ #266 ] pilorama: Allow to get current tree height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
04727ce1d6
Revert "[ #135 ] signature: Add tracing"
...
This reverts commit 5778980252
2023-04-12 16:57:02 +03:00
08769f413f
Revert "[ #135 ] acl: Add tracing spans"
...
This reverts commit b2ca730547
2023-04-12 16:54:13 +03:00
5d2affa5cd
testutil: Fix linter warning
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 11:00:02 +03:00
5778980252
[ #135 ] signature: Add tracing
...
ci/woodpecker/push/pre-commit Pipeline was successful
Add tracing to verify request and sign response.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
b2ca730547
[ #135 ] acl: Add tracing spans
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
0920d848d0
[ #135 ] get-object: Add tracing spans
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
5af9f58469
[ #135 ] tracing: Add tracing to node gRPC endpoints
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
c4865783fc
[ #236 ] blobstor/test: Prefill storage in parallel in read benchmark
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:11 +03:00
6ad5c38225
[ #236 ] testutil: Use random object id in RandObjGenerator
...
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8 227425 4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
c85a0bc866
[ #236 ] blobstor/test: Reduce test descriptions
...
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
93eba19a8e
[ #223 ] objectsvc: Refactor split-tree traverse
...
Resolve funlen & gocognit linters for traverseSplitChain method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
2ed9fd3f94
[ #223 ] objectsvc: Refactor request parameters
...
Resolve containedctx linter for commonPrm.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ccf8463e69
[ #223 ] controlsvc: Drop unnecessary nolint
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ae86cda58c
[ #223 ] sg: Refactor storage group parameters
...
Resolve containedctx linter for SearchSGPrm and GetSGPrm structs.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
6016d78a45
[ #223 ] core: Refactor object format validator
...
Resolve funlen linter for FormatValidator.ValidateContent method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 13:56:09 +03:00
02831d427b
[ #213 ] metrics: Refactor object metrics
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Resolve funlen linter for newObjectServiceMetrics function.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
38ae71cc7d
[ #213 ] metrics: Refactor engine metrics
...
Resolve funlen linter for newEngineMetrics function.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
0b9622c418
[ #191 ] control: Add Doctor RPC
...
Doctor RPC performs complex operations on the storage engine.
Currently only duplicate removal is supported.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
dbc3811ff4
[ #191 ] engine: Allow to remove redundant object copies
...
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
cb172e73a6
[ #228 ] node: Use uber atomic package instead standard
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 15:37:27 +00:00
c236b54a65
[ #212 ] reputationsvc: Resolve funlen linter
...
ci/woodpecker/push/pre-commit Pipeline was successful
Resolve funlen linter for Calculator.iterateDaughter method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
7ebbfa3358
[ #212 ] reputationsvc: Resolve linters and rename
...
Resolved containedctx linters.
Renamed context structs and interfaces to more understandble names.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
469e8a6e59
[ #212 ] reputationsvc: Resolve containedctx linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
e2f13d03d7
[ #222 ] auditsvc: Refactor PoR audit
...
ci/woodpecker/push/pre-commit Pipeline was successful
Resolve funlen linter for Context.checkStorageGroupPoR method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
e8d340287f
[ #222 ] auditsvc: Refactor audit task
...
Resolve containedctx linter. Cancel task by listen cancel.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
3dbff0a478
[ #222 ] auditsvc: Resolve containedctx linter
...
Resolve containedctx linter for commonCommunicatorPrm type.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
fe87735073
[ #219 ] morph: Refactor notary preparator
...
ci/woodpecker/push/pre-commit Pipeline was successful
Resolve funlen linter for Preparator.Prepare method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
d07e40d6fe
[ #219 ] morph: Refactor moprh event listener
...
Resolve funlen and gocognit linters for listener.listenLoop method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
775179f823
[ #219 ] morph: Refactor notary invoke
...
Resolve funlen linter for Client.notaryInvoke method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
e815b19101
[ #219 ] morph: Resolve containedctx linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
56282edf02
[ #166 ] node: Parallelize background tree service sync
...
ci/woodpecker/push/pre-commit Pipeline was successful
* Run sync task for nodes in parallel within errgroup worker pool
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 04:43:32 +00:00
4ec69cbbf8
[ #203 ] control: Fix test
...
Fix shard compare loop.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
f32f61df87
[ #203 ] pilorama: Refactor tests
...
Do not pass 0 as channel capacity.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
8908798f59
[ #203 ] node: Resolve unused vars
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
bab11492ad
[ #203 ] node: Resolve never used errors
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
68a2f36636
[ #203 ] morph: Mark depracated methods
...
Skip staticcheck for depracated methods. Will be fixed soon.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
9e2df4b7c7
[ #203 ] node: Fix double imports
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
ab891517de
[ #116 ] node: Fix bug with extra generated files in TestReload
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
* Create testNewEngine in engineWithShards without default opts
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 18:49:34 +03:00
c58ab0c369
[ #193 ] getsvc: Reduce private key requests
...
ci/woodpecker/push/pre-commit Pipeline was successful
Get private key only once for request forwaring.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
89924071cd
[ #193 ] getsvc: Edit request forwarder signature
...
Pass context to forwarder direct, without closure.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
