25bedab91a
[ #425 ] object: Do not store large slices in pool
...
Dynamically growing an unbounded buffers can cause a large
amount of memory to be pinned and never be freed.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-02 17:48:24 +03:00
5e2fcec60f
[ #396 ] treesvc: properly remember last height on shutdown
...
Previously `newHeight` was updated in parallel, so that applying
operation at height H did not imply successful TreeApply() for H-1.
And because we have no context in TreeUpdateLastSyncHeight(), invalid
starting height could be written if the context was canceled.
In this commit we return the new height only if all operations were
successfully applied.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-29 10:25:26 +00:00
139ded93e1
[ #396 ] pilorama: Disallow applying same operations
...
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-29 10:25:26 +00:00
f866ec1399
[ #392 ] gc: Use defer to mark handler done
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 15:52:49 +03:00
a506da97d6
[ #384 ] shard: Add unit test
...
Add test to check that oject not found error will be returned,
if object doesn't exist in blobstore.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 09:55:16 +03:00
1dd84eef77
[ #384 ] shard: Cancel GC if change mode requested
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 09:55:16 +03:00
9bda6e0b8b
[ #332 ] gc: Fix expired complex object deletion
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-18 12:51:34 +00:00
ceb9deb7f1
[ #337 ] morph: Move subscription logic to subscriber
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-18 11:28:50 +03:00
493cafc62a
[ #355 ] Increase tree svc client cache size to test hypotheses
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 15:17:35 +03:00
3711976dfc
[ #314 ] writecache: remove objects right after they are flushed
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-16 14:16:25 +03:00
Pavel Karpy
c3f5045842
[ #314 ] wc: Do not lose small objects on disk errors
...
Do return error if an object could not been stored on WC's disk.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-15 16:27:44 +03:00
Pavel Karpy
ab65063d6d
[ #314 ] wc: Simplify background workers naming
...
Also, drop not used arg.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-15 16:27:42 +03:00
Pavel Karpy
c60029d3b0
[ #323 ] node: Fix tree svc panic
...
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-04 20:04:30 +03:00
Pavel Karpy
0beb7ccf5c
[ #284 ] node: Use copy_number on server side
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-26 10:57:34 +03:00
79d59e4ed2
[ #266 ] services/tree: Do not accept requests until initial sync is finished
...
`Apply` is deliberately left out -- we don't want to miss anything new.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
364b4ac572
[ #266 ] services/tree: Batch operations on the service level
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
f7679a8168
[ #266 ] services/tree: Return operation log up to some height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
2dc2fe8780
[ #266 ] pilorama: Allow to get current tree height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
04727ce1d6
Revert "[ #135 ] signature: Add tracing"
...
This reverts commit 5778980252
2023-04-12 16:57:02 +03:00
08769f413f
Revert "[ #135 ] acl: Add tracing spans"
...
This reverts commit b2ca730547
2023-04-12 16:54:13 +03:00
5d2affa5cd
testutil: Fix linter warning
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 11:00:02 +03:00
5778980252
[ #135 ] signature: Add tracing
...
Add tracing to verify request and sign response.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
b2ca730547
[ #135 ] acl: Add tracing spans
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
0920d848d0
[ #135 ] get-object: Add tracing spans
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
5af9f58469
[ #135 ] tracing: Add tracing to node gRPC endpoints
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
c4865783fc
[ #236 ] blobstor/test: Prefill storage in parallel in read benchmark
...
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:11 +03:00
6ad5c38225
[ #236 ] testutil: Use random object id in RandObjGenerator
...
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8 227425 4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
c85a0bc866
[ #236 ] blobstor/test: Reduce test descriptions
...
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
93eba19a8e
[ #223 ] objectsvc: Refactor split-tree traverse
...
Resolve funlen & gocognit linters for traverseSplitChain method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
2ed9fd3f94
[ #223 ] objectsvc: Refactor request parameters
...
Resolve containedctx linter for commonPrm.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ccf8463e69
[ #223 ] controlsvc: Drop unnecessary nolint
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ae86cda58c
[ #223 ] sg: Refactor storage group parameters
...
Resolve containedctx linter for SearchSGPrm and GetSGPrm structs.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
6016d78a45
[ #223 ] core: Refactor object format validator
...
Resolve funlen linter for FormatValidator.ValidateContent method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 13:56:09 +03:00
02831d427b
[ #213 ] metrics: Refactor object metrics
...
Resolve funlen linter for newObjectServiceMetrics function.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
38ae71cc7d
[ #213 ] metrics: Refactor engine metrics
...
Resolve funlen linter for newEngineMetrics function.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
0b9622c418
[ #191 ] control: Add Doctor RPC
...
Doctor RPC performs complex operations on the storage engine.
Currently only duplicate removal is supported.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
dbc3811ff4
[ #191 ] engine: Allow to remove redundant object copies
...
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
cb172e73a6
[ #228 ] node: Use uber atomic package instead standard
...
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 15:37:27 +00:00
c236b54a65
[ #212 ] reputationsvc: Resolve funlen linter
...
Resolve funlen linter for Calculator.iterateDaughter method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
7ebbfa3358
[ #212 ] reputationsvc: Resolve linters and rename
...
Resolved containedctx linters.
Renamed context structs and interfaces to more understandble names.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
469e8a6e59
[ #212 ] reputationsvc: Resolve containedctx linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
e2f13d03d7
[ #222 ] auditsvc: Refactor PoR audit
...
Resolve funlen linter for Context.checkStorageGroupPoR method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
e8d340287f
[ #222 ] auditsvc: Refactor audit task
...
Resolve containedctx linter. Cancel task by listen cancel.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
3dbff0a478
[ #222 ] auditsvc: Resolve containedctx linter
...
Resolve containedctx linter for commonCommunicatorPrm type.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
fe87735073
[ #219 ] morph: Refactor notary preparator
...
Resolve funlen linter for Preparator.Prepare method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
d07e40d6fe
[ #219 ] morph: Refactor moprh event listener
...
Resolve funlen and gocognit linters for listener.listenLoop method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
775179f823
[ #219 ] morph: Refactor notary invoke
...
Resolve funlen linter for Client.notaryInvoke method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
e815b19101
[ #219 ] morph: Resolve containedctx linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
56282edf02
[ #166 ] node: Parallelize background tree service sync
...
* Run sync task for nodes in parallel within errgroup worker pool
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 04:43:32 +00:00
4ec69cbbf8
[ #203 ] control: Fix test
...
Fix shard compare loop.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
