Sync flag in tree service API #1611

New issue

Open

opened 2025-01-23 06:55:32 +00:00 by alexvanin · 0 comments

alexvanin commented

2025-01-23 06:55:32 +00:00

Owner

S3 gateway stores some bucket (container) configurations in a tree service. Tree service is not synchronously replicated, thus in order to be consistent, gateway now picks a "container leader" and sends all read and write operation to the leader node, if available. See frostfs-sdk-go#305

During network connectivity split, leader may not be available. This is okay for regular data-path operations to store and receive object. Objects going to be available after connection is reestablished.

However it is not okay for bucket configuration, especially for "versioning" setting, because it changes the way how S3 gateway stores meta data about the object in tree service. Different network parts may see different "versioning" setting, which is not so great.

Describe the solution you'd like

Allow client to enable synchronous replication for some tree service operations. It may be sync flag in request body. When storage node sees sync flag, it does not return response until tree node is synced across all container nodes.

Describe alternatives you've considered

We discussed an option to store such sensetive settings in a blockchain, see frostfs-contract#143

Also we discussed a possibility for S3 gateway to store all objects the same way regardless of "versioning" status, and process "versioning" status only for listing / get operations. It may produce other issues such as:

unclear behaviour on delete operations,
unclear metrics when user expects one object in unversioned bucket but gets many objects on storage level.

## Is your feature request related to a problem? Please describe. S3 gateway stores some bucket (container) configurations in a tree service. Tree service is not synchronously replicated, thus in order to be consistent, gateway now picks a "container leader" and sends all read and write operation to the leader node, if available. See [frostfs-sdk-go#305](https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pulls/305) During network connectivity split, leader may not be available. This is okay for regular data-path operations to store and receive object. Objects going to be available after connection is reestablished. However it is not okay for bucket configuration, especially for "versioning" setting, because it changes the way how S3 gateway stores meta data about the object in tree service. Different network parts may see different "versioning" setting, which is not so great. ## Describe the solution you'd like Allow client to enable synchronous replication for some tree service operations. It may be `sync` flag in request body. When storage node sees `sync` flag, it does not return response until tree node is synced across all container nodes. ## Describe alternatives you've considered We discussed an option to store such sensetive settings in a blockchain, see [frostfs-contract#143](https://git.frostfs.info/TrueCloudLab/frostfs-contract/issues/145) Also we discussed a possibility for S3 gateway to store all objects the same way regardless of "versioning" status, and process "versioning" status only for listing / get operations. It may produce other issues such as: - unclear behaviour on delete operations, - unclear metrics when user expects one object in unversioned bucket but gets many objects on storage level.