5e2fcec60f
[ #396 ] treesvc: properly remember last height on shutdown
...
ci/woodpecker/push/pre-commit Pipeline was successful
Previously `newHeight` was updated in parallel, so that applying
operation at height H did not imply successful TreeApply() for H-1.
And because we have no context in TreeUpdateLastSyncHeight(), invalid
starting height could be written if the context was canceled.
In this commit we return the new height only if all operations were
successfully applied.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-29 10:25:26 +00:00
79d59e4ed2
[ #266 ] services/tree: Do not accept requests until initial sync is finished
...
ci/woodpecker/pr/pre-commit Pipeline failed
ci/woodpecker/push/pre-commit Pipeline failed
`Apply` is deliberately left out -- we don't want to miss anything new.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
364b4ac572
[ #266 ] services/tree: Batch operations on the service level
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-18 13:30:45 +03:00
5af9f58469
[ #135 ] tracing: Add tracing to node gRPC endpoints
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
56282edf02
[ #166 ] node: Parallelize background tree service sync
...
ci/woodpecker/push/pre-commit Pipeline was successful
* Run sync task for nodes in parallel within errgroup worker pool
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 04:43:32 +00:00
9e2df4b7c7
[ #203 ] node: Fix double imports
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
5368c4207a
[ #156 ] services/tree: Split syncLoop() in functions
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
47e8c5bf23
[ #156 ] pilorama: Remove CIDDescriptor from TreeApply()
...
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
only used in controlled environments.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
481a1ca6f3
[ #148 ] linter: Add gocognit linter
...
Code with high cognitive complexity is hard intuitively to understand
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
97c36ed3ec
[ #148 ] linter: Add funlen linter
...
Long functions are hard to understand and source of errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
6cd806f998
[ #82 ] services/tree: Save last synchronized height in a persistent storage
...
Remember the last synchronized height and use it after service restart.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
f2250a316f
[ #129 ] tree: Do not remove tree if the netmap is empty
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
46c62be7e8
[ #28 ] Fix linter issues
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
b4e90cdf51
[ #2165 ] pilorama: Optimize TreeApply when used for synchronization
...
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f9fcd85363
[ #2165 ] services/tree: Remember starting height for the synchronization
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
06137dbf8e
[ #2165 ] services/tree: Do not export synchronizeAllTrees
...
It is used only privately.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c299b98afe
[ #2165 ] services/tree: Parallelize synchronization
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
de9957e076
[ #2165 ] services/tree: Always synchronize all containers
...
In case of split-brain we must synchronize everything.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
9e0decd12d
[ #2162 ] services/tree: Close connection after the syncronization
...
There was a goroutine leak here.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
3bb5a320d7
[ #2154 ] services/tree: Do not log an error when synchronizing container of 1 node
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
387d1e2977
[ #2127 ] services/tree: Randomize node order for synchronization
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a
Move to frostfs-node
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
9a20498f34
[ #1940 ] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-19 11:01:04 +03:00
Pavel Karpy
e1be0180f6
[ #1329 ] tree: Sync trees when a node first time appears in a container
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-21 18:47:56 +03:00
Pavel Karpy
1766ca2039
[ #1902 ] tree: Allow synchronize all the container trees
...
Add `SynchronizeAllTrees` method of the Tree service. It allows fetching
tree IDs and sync all of them. Share common logic b/w the new method and
the `SynchronizeTree`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
bc67f77d86
[ #1329 ] tree: Fix sync applying
...
Add the node position in a container and the container size to the CID
descriptor that is passed to the `TreeApply`. Previously, `checkValid` does
not allow any log operarations.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
f5f560d903
[ #1329 ] tree: Do not sync trees that are from external containers
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
13c4a9f4b8
[ #1332 ] tree: Make SignMessage public
...
It will allow reusing signing routine in other components
(e.g. `neofs-cli`).
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Evgenii Stratonikov
8a47f9ba12
[ #1753 ] services/tree: Remove MaxGetSubTreeCount constant
...
It was used only in `GetOpLog` and ignored on the server side.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
4558f30575
[ #1644 ] services/tree: Do not use deprecated gRPC options
...
Get rid of `grpc.WithInsecure`.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
39f47f61c6
[ #1445 ] services/tree: Cache the list of container nodes
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113
[ #1442 ] pilorama: Generate timestamp based on node position in the container
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f4bc1c601a
[ #1329 ] services/tree: Synchronize from the last stored height
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
