Try to adopt tree service to get rid of object searches #4

New issue

Open

opened 2024-12-26 10:48:04 +00:00 by alexvanin · 2 comments

alexvanin commented

2024-12-26 10:48:04 +00:00

Owner

See adoption of tree service in s3-gateway (see SDK Tree Pool) and try to adopt the same mechanism for restic. This should replace all Object.Search operations.

See adoption of tree service in s3-gateway (see [SDK Tree Pool](https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/src/branch/master/pool/tree)) and try to adopt the same mechanism for restic. This should replace all Object.Search operations.

KurlesHS was assigned by alexvanin

2024-12-26 10:48:04 +00:00

alexvanin added the

discussion

label 2024-12-26 10:51:56 +00:00

alexvanin referenced this issue from TrueCloudLab/rclone

2024-12-26 10:52:41 +00:00

Try to adopt tree service to get rid of object searches #2

alexvanin commented

2024-12-26 10:54:49 +00:00

Author

Owner

@realloc has some consideration about the usage of tree service in restic/rclone, let's discuss it first.

realloc commented

2024-12-27 08:48:22 +00:00

Owner

To integrate with rclone, restic, and similar services, I'd prefer not to rely on a Tree Service.

First of all, the Tree Service is not yet part of the main API, and we are planning a significant overhaul in the nearest future.

Secondly, we would like to utilize rclone for scenarios that involve uploading vast amounts of immutable data without a changing hierarchy. Examples include test reports, build results, package repositories, and so on. In these scenarios, the tree structure can become a bottleneck, also leading to unnecessary overhead.

Regarding restic, it would be ideal, if technically feasible, to structure everything in a way that allows direct access to objects by their addresses, bypassing the search and tree access stages. We are considering developing a mechanism to accomplish this without involving SEARCH in the future, which further supports the argument against relying on the Tree Service in this context.

To integrate with rclone, restic, and similar services, I'd prefer not to rely on a Tree Service. First of all, the Tree Service is not yet part of the main API, and we are planning a significant overhaul in the nearest future. Secondly, we would like to utilize rclone for scenarios that involve uploading vast amounts of immutable data without a changing hierarchy. Examples include test reports, build results, package repositories, and so on. In these scenarios, the tree structure can become a bottleneck, also leading to unnecessary overhead. Regarding restic, it would be ideal, if technically feasible, to structure everything in a way that allows direct access to objects by their addresses, bypassing the search and tree access stages. We are considering developing a mechanism to accomplish this without involving SEARCH in the future, which further supports the argument against relying on the Tree Service in this context.

🚀 1