TrueCloudLab/frostfs-node
19
1
Fork 27
Code Issues 91 Pull requests 7 Projects Releases 1 Activity Actions
4048 commits 8 branches 81 tags 52 MiB
Commit graph

39 commits

Author SHA1 Message Date
Dmitrii Stepanov
957a43a124 [#266] services/tree: Add sync check
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Do not accept requests until initial sync is finished.
`Apply` is deliberately left out -- we don't want to miss anything new.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-13 10:00:45 +00:00
Dmitrii Stepanov
dcdfb6ed41 [#412] node: Use observability interceptors 
Use metrics and tracing interceptors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Alejandro Lopez
bc34fee6a7 [#370] Add tree service metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-26 13:39:12 +00:00
Dmitrii Stepanov
529d0bc710 [#302] tree: Drop unused ctx
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-02 16:54:06 +03:00
Airat Arifullin
9d01029733 [#166] node: Parallelize background tree service sync by batching
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
* Merge operations

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-26 10:17:56 +00:00
Airat Arifullin
299b24b974 [#166] node: Parallelize background tree service sync by batching 
* Concurrently dispatch TreeApply operations for batching in forest

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-26 10:17:56 +00:00
Dmitrii Stepanov
6121b541b5 [#242] treesvc: Add tracing spans
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Evgenii Stratonikov
0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Dmitrii Stepanov
5af9f58469 [#135] tracing: Add tracing to node gRPC endpoints 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Airat Arifullin
56282edf02 [#166] node: Parallelize background tree service sync
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
* Run sync task for nodes in parallel within errgroup worker pool

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 04:43:32 +00:00
Dmitrii Stepanov
9e2df4b7c7 [#203] node: Fix double imports 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Evgenii Stratonikov
5368c4207a [#156] services/tree: Split syncLoop() in functions 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov
47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply() 
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Dmitrii Stepanov
481a1ca6f3 [#148] linter: Add gocognit linter 
Code with high cognitive complexity is hard intuitively to understand

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov
97c36ed3ec [#148] linter: Add funlen linter 
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Evgenii Stratonikov
6cd806f998 [#82] services/tree: Save last synchronized height in a persistent storage 
Remember the last synchronized height and use it after service restart.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 11:25:44 +00:00
Evgenii Stratonikov
f2250a316f [#129] tree: Do not remove tree if the netmap is empty 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-10 10:59:15 +00:00
Alex Vanin
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Evgenii Stratonikov
46c62be7e8 [#28] Fix linter issues 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
b4e90cdf51 [#2165] pilorama: Optimize TreeApply when used for synchronization 
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
f9fcd85363 [#2165] services/tree: Remember starting height for the synchronization 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
06137dbf8e [#2165] services/tree: Do not export synchronizeAllTrees 
It is used only privately.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
c299b98afe [#2165] services/tree: Parallelize synchronization 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
de9957e076 [#2165] services/tree: Always synchronize all containers 
In case of split-brain we must synchronize everything.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
9e0decd12d [#2162] services/tree: Close connection after the syncronization 
There was a goroutine leak here.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
3bb5a320d7 [#2154] services/tree: Do not log an error when synchronizing container of 1 node 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
387d1e2977 [#2127] services/tree: Randomize node order for synchronization 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Anton Nikiforov
9a20498f34 [#1940] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2022-11-19 11:01:04 +03:00
Pavel Karpy
e1be0180f6 [#1329] tree: Sync trees when a node first time appears in a container 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-21 18:47:56 +03:00
Pavel Karpy
1766ca2039 [#1902] tree: Allow synchronize all the container trees 
Add `SynchronizeAllTrees` method of the Tree service. It allows fetching
tree IDs and sync all of them. Share common logic b/w the new method and
the `SynchronizeTree`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Pavel Karpy
bc67f77d86 [#1329] tree: Fix sync applying 
Add the node position in a container and the container size to the CID
descriptor that is passed to the `TreeApply`. Previously, `checkValid` does
not allow any log operarations.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-17 20:53:34 +03:00
Pavel Karpy
f5f560d903 [#1329] tree: Do not sync trees that are from external containers 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-17 20:53:34 +03:00
Pavel Karpy
13c4a9f4b8 [#1332] tree: Make SignMessage public 
It will allow reusing signing routine in other components
(e.g. `neofs-cli`).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-13 20:01:48 +03:00
Evgenii Stratonikov
8a47f9ba12 [#1753] services/tree: Remove MaxGetSubTreeCount constant 
It was used only in `GetOpLog` and ignored on the server side.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
4558f30575 [#1644] services/tree: Do not use deprecated gRPC options 
Get rid of `grpc.WithInsecure`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
39f47f61c6 [#1445] services/tree: Cache the list of container nodes 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113 [#1442] pilorama: Generate timestamp based on node position in the container 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f4bc1c601a [#1329] services/tree: Synchronize from the last stored height 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00