TrueCloudLab/frostfs-node
20
1
Fork 27
Code Issues 92 Pull requests 4 Projects Releases 1 Activity Actions
4873 commits 8 branches 85 tags 54 MiB
Commit graph

154 commits

Author SHA1 Message Date
Evgenii Stratonikov
c299b98afe [#2165] services/tree: Parallelize synchronization 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
de9957e076 [#2165] services/tree: Always synchronize all containers 
In case of split-brain we must synchronize everything.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
9e0decd12d [#2162] services/tree: Close connection after the syncronization 
There was a goroutine leak here.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy
306609030a [#2159] node: Add tree replication timeout configuration 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
3bb5a320d7 [#2154] services/tree: Do not log an error when synchronizing container of 1 node 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
387d1e2977 [#2127] services/tree: Randomize node order for synchronization 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Anton Nikiforov
9a20498f34 [#1940] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2022-11-19 11:01:04 +03:00
Pavel Karpy
e1be0180f6 [#1329] tree: Sync trees when a node first time appears in a container 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-21 18:47:56 +03:00
Pavel Karpy
1766ca2039 [#1902] tree: Allow synchronize all the container trees 
Add `SynchronizeAllTrees` method of the Tree service. It allows fetching
tree IDs and sync all of them. Share common logic b/w the new method and
the `SynchronizeTree`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Pavel Karpy
6d4beea187 [#1902] tree: Extend grpc service with ListTrees method 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Pavel Karpy
3aa9938b8f [#1902] Update protoc to v3.21.7 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Pavel Karpy
bc67f77d86 [#1329] tree: Fix sync applying 
Add the node position in a container and the container size to the CID
descriptor that is passed to the `TreeApply`. Previously, `checkValid` does
not allow any log operarations.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-17 20:53:34 +03:00
Pavel Karpy
f5f560d903 [#1329] tree: Do not sync trees that are from external containers 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-17 20:53:34 +03:00
Pavel Karpy
13c4a9f4b8 [#1332] tree: Make SignMessage public 
It will allow reusing signing routine in other components
(e.g. `neofs-cli`).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-13 20:01:48 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component 
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-12 18:11:05 +03:00
Anton Nikiforov
feaa9eace7 [#1768] Add healthcheck method to the Tree service 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2022-10-11 12:53:45 +03:00
Leonard Lyubich
485a5418d2 [#1793] node: Serve NetmapService.NetmapSnapshot RPC 
There is no more need to serve the same request on Control API.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-09-27 09:03:17 +03:00
Leonard Lyubich
ae655b74f0 *: Exec make fmts 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-09-14 17:52:49 +04:00
Pavel Karpy
4083a62679 [#1628] tree: Pre-allocate errors in ACL checks 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy
5adf089c1d [#1628] tree: Log unacceptable bearer attachment 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy
2ffcd02ac3 [#1628] tree: Document ACL checks in tree service 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy
8d0906c6ab [#1628] tree: Skip eACL filters for tree requests 
Do not call `CalculateAction` for the eACL checks since it requires object
headers that are meaningless in the tree context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy
876e014b5d [#1628] tree: Make ACL checks the same way as for object requests 
1. Do not require a request to be signed by the container owner if a
bearer token is missing
2. Do not check the system role since public requests are not expected to
be signed by IR or a container node (unlike the object requests)

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Evgenii Stratonikov
9da5d784cb [#1630] neofs-node: Remove trees on container removal event 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-12 09:54:15 +03:00
Evgenii Stratonikov
0140ac354b [#1759] services/tree: Make logs more descriptive 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 12:57:33 +04:00
Evgenii Stratonikov
74c861342e [#1753] services/tree: Add parent ID to the GetNodeByPath response 
Currently, you need to use `GetSubTree` to get parent, which seems an
overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
8a47f9ba12 [#1753] services/tree: Remove MaxGetSubTreeCount constant 
It was used only in `GetOpLog` and ignored on the server side.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
9840936a4f [#1753] services/tree: Do not restrict depth in GetSubTree 
Previously, the depth was restricted because with BFS the amount of
nodes we have in memory blows up exponentially. With DFS is is linear,
so we can process trees of arbitrary depth.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
4558f30575 [#1644] services/tree: Do not use deprecated gRPC options 
Get rid of `grpc.WithInsecure`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
10c855efef [#1624] go.mod: Update dependencies 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
7e717e80ef [#1623] services/tree: Fix error message 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-22 15:13:03 +03:00
Evgenii Stratonikov
b549cc314c [#1607] services/tree: allow to customize some parameters 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f80e52fbea [#1334] services/tree: Document *.proto files 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5e843a73f9 [#1333] services/control: Return pilorama info in ListShards RPC 
Do not return backend type from the service for now, because memory
backend is expected to vanish.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5ffbeb76e6 [#1456] services/tree: Wait some time before reconnecting after failure 
In case node is down or failing for some reason, we can expect `Dial` to
fail. In case we actively try to replicate and `Dial` always takes 2
seconds, replication-related channels quickly become full. That affects
latency of all other write operations.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
39f47f61c6 [#1445] services/tree: Cache the list of container nodes 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a2edfec0c3 [#1446] services/tree: Use grpc.WithInsecure only for nodes without TLS 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
33d8fb187a [#1427] services/tree: Parallelize replicator 
Before this commit the replication channel was quickly filled under
heavy load. This lead to the continuously increasing latency for all
write operations. Now it looks better.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b04f712773 [#1446] services/tree: Cache connections to the container nodes 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113 [#1442] pilorama: Generate timestamp based on node position in the container 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3caa982283 [#1442] services/tree: Fix log message for failed Apply 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
01d2e06a9b [#1401] services/tree: Marshal public key once 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
1f5a650b05 [#1401] services/tree: Retransmit queries to container nodes 
Also fix a bug with replicator using the multiaddress instead of
<host>:<port> format expected by gRPC library.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f4bc1c601a [#1329] services/tree: Synchronize from the last stored height 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
536857ea5a [#1329] services/tree: Implement GetOpLog RPC 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
886baf3136 [#1426] services/tree: Do not replicate to a local node 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad48918a97 [#1406] pilorama: Return parent from TreeGetMeta 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Denis Kirillov
b30f14978d [#1404] services/tree: Fix nodeId in GetSubTree 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
45321b3f6d [#1328] services/tree: Fix grpc import path 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
16e3421825 [#1328] services/tree: Implement access control 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
94b9e13431 [#1328] services/tree: Fix proto field numbers 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
aea855e8f3 [#1326] services/tree: Implement GetSubTree RPC 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
62154da17c [#1324] services/tree: Implement Object Tree Service 
Object Tree Service allows changing trees assotiated with
the container in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00