TrueCloudLab
/
frostfs-node
18
1
Fork 20
Code Issues 88 Pull Requests 7 Actions Releases Activity
3,715 Commits
7 Branches
20 Tags
47 MiB
Commit Graph

2438 Commits (f152f5d553fe5d6013640d5d89e63654f077ce15)

Author SHA1 Message Date
Dmitrii Stepanov 1cae03c47c [#452] engine: Set Disabled mode to deleted shard
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-19 19:04:06 +03:00
Dmitrii Stepanov 25bedab91a [#425] object: Do not store large slices in pool
ci/woodpecker/pr/pre-commit Pipeline was successful Details 
Dynamically growing an unbounded buffers can cause a large
amount of memory to be pinned and never be freed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-02 17:48:24 +03:00
Evgenii Stratonikov 5e2fcec60f [#396] treesvc: properly remember last height on shutdown
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Previously `newHeight` was updated in parallel, so that applying
operation at height H did not imply successful TreeApply() for H-1.
And because we have no context in TreeUpdateLastSyncHeight(), invalid
starting height could be written if the context was canceled.

In this commit we return the new height only if all operations were
successfully applied.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-29 10:25:26 +00:00
Evgenii Stratonikov 139ded93e1 [#396] pilorama: Disallow applying same operations 
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
   overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-29 10:25:26 +00:00
Dmitrii Stepanov f866ec1399 [#392] gc: Use defer to mark handler done 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-25 15:52:49 +03:00
Dmitrii Stepanov a506da97d6 [#384] shard: Add unit test
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Add test to check that oject not found error will be returned,
if object doesn't exist in blobstore.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-25 09:55:16 +03:00
Dmitrii Stepanov 1dd84eef77 [#384] shard: Cancel GC if change mode requested 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-25 09:55:16 +03:00
Dmitrii Stepanov 9bda6e0b8b [#332] gc: Fix expired complex object deletion
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-18 12:51:34 +00:00
Evgenii Stratonikov ceb9deb7f1 [#337] morph: Move subscription logic to subscriber
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-18 11:28:50 +03:00
Alejandro Lopez 493cafc62a [#355] Increase tree svc client cache size to test hypotheses
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-17 15:17:35 +03:00
Evgenii Stratonikov 3711976dfc [#314] writecache: remove objects right after they are flushed
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-16 14:16:25 +03:00
Pavel Karpy c3f5045842 [#314] wc: Do not lose small objects on disk errors 
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-15 16:27:44 +03:00
Pavel Karpy ab65063d6d [#314] wc: Simplify background workers naming 
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-15 16:27:42 +03:00
Pavel Karpy c60029d3b0 [#323] node: Fix tree svc panic
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-04 20:04:30 +03:00
Pavel Karpy 0beb7ccf5c [#284] node: Use `copy_number` on server side
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-26 10:57:34 +03:00
Evgenii Stratonikov 79d59e4ed2 [#266] services/tree: Do not accept requests until initial sync is finished
ci/woodpecker/pr/pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline failed Details 
`Apply` is deliberately left out -- we don't want to miss anything new.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-18 13:30:45 +03:00
Evgenii Stratonikov 364b4ac572 [#266] services/tree: Batch operations on the service level 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-18 13:30:45 +03:00
Evgenii Stratonikov f7679a8168 [#266] services/tree: Return operation log up to some height 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-18 13:30:45 +03:00
Evgenii Stratonikov 2dc2fe8780 [#266] pilorama: Allow to get current tree height 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-18 13:30:45 +03:00
Evgenii Stratonikov 04727ce1d6 Revert "[#135] signature: Add tracing" 
This reverts commit 5778980252.
 2023-04-12 16:57:02 +03:00
Evgenii Stratonikov 08769f413f Revert "[#135] acl: Add tracing spans" 
This reverts commit b2ca730547.
 2023-04-12 16:54:13 +03:00
Evgenii Stratonikov 5d2affa5cd testutil: Fix linter warning 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-12 11:00:02 +03:00
Dmitrii Stepanov 5778980252 [#135] signature: Add tracing
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Add tracing to verify request and sign response.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Dmitrii Stepanov b2ca730547 [#135] acl: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Dmitrii Stepanov 0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Dmitrii Stepanov 5af9f58469 [#135] tracing: Add tracing to node gRPC endpoints 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Evgenii Stratonikov c4865783fc [#236] blobstor/test: Prefill storage in parallel in read benchmark
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:11 +03:00
Evgenii Stratonikov 6ad5c38225 [#236] testutil: Use random object id in `RandObjGenerator` 
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8            227425              4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8     --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8            --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:04 +03:00
Evgenii Stratonikov c85a0bc866 [#236] blobstor/test: Reduce test descriptions 
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:04 +03:00
Dmitrii Stepanov 93eba19a8e [#223] objectsvc: Refactor split-tree traverse 
Resolve funlen & gocognit linters for traverseSplitChain method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 14:16:34 +03:00
Dmitrii Stepanov 2ed9fd3f94 [#223] objectsvc: Refactor request parameters 
Resolve containedctx linter for commonPrm.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 14:16:34 +03:00
Dmitrii Stepanov ccf8463e69 [#223] controlsvc: Drop unnecessary nolint 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 14:16:34 +03:00
Dmitrii Stepanov ae86cda58c [#223] sg: Refactor storage group parameters 
Resolve containedctx linter for SearchSGPrm and GetSGPrm  structs.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 14:16:34 +03:00
Dmitrii Stepanov 6016d78a45 [#223] core: Refactor object format validator 
Resolve funlen linter for FormatValidator.ValidateContent method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 13:56:09 +03:00
Dmitrii Stepanov 02831d427b [#213] metrics: Refactor object metrics
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for newObjectServiceMetrics function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 09:43:24 +03:00
Dmitrii Stepanov 38ae71cc7d [#213] metrics: Refactor engine metrics 
Resolve funlen linter for newEngineMetrics function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 09:43:24 +03:00
Evgenii Stratonikov 0b9622c418 [#191] control: Add Doctor RPC 
Doctor RPC performs complex operations on the storage engine.
Currently only duplicate removal is supported.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-07 17:25:50 +00:00
Evgenii Stratonikov dbc3811ff4 [#191] engine: Allow to remove redundant object copies 
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-07 17:25:50 +00:00
Airat Arifullin cb172e73a6 [#228] node: Use uber atomic package instead standard
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 15:37:27 +00:00
Dmitrii Stepanov c236b54a65 [#212] reputationsvc: Resolve funlen linter
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for Calculator.iterateDaughter method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov 7ebbfa3358 [#212] reputationsvc: Resolve linters and rename 
Resolved containedctx linters.
Renamed context structs and interfaces to more understandble names.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov 469e8a6e59 [#212] reputationsvc: Resolve containedctx linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov e2f13d03d7 [#222] auditsvc: Refactor PoR audit
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for Context.checkStorageGroupPoR method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Dmitrii Stepanov e8d340287f [#222] auditsvc: Refactor audit task 
Resolve containedctx linter. Cancel task by listen cancel.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Dmitrii Stepanov 3dbff0a478 [#222] auditsvc: Resolve containedctx linter 
Resolve containedctx linter for commonCommunicatorPrm type.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Dmitrii Stepanov fe87735073 [#219] morph: Refactor notary preparator
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for Preparator.Prepare method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:43:20 +03:00
Dmitrii Stepanov d07e40d6fe [#219] morph: Refactor moprh event listener 
Resolve funlen and gocognit linters for listener.listenLoop method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:43:20 +03:00
Dmitrii Stepanov 775179f823 [#219] morph: Refactor notary invoke 
Resolve funlen linter for Client.notaryInvoke method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:43:20 +03:00
Dmitrii Stepanov e815b19101 [#219] morph: Resolve containedctx linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:43:20 +03:00
Airat Arifullin 56282edf02 [#166] node: Parallelize background tree service sync
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Run sync task for nodes in parallel within errgroup worker pool

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 04:43:32 +00:00