Object expiration data in the tree service #141

New Issue

Open

opened 2023-06-14 14:04:43 +00:00 by alexvanin · 4 comments

alexvanin commented

2023-06-14 14:04:43 +00:00

Owner

S3 Gateway provides a possibility to set an expiration for the objects. However the tree service and object service are not synced in the storage node. So expired object will not be removed from tree service, which may lead to unexpected listing results.

Describe the solution you'd like

Store expiration epoch of the object in the tree and ignore expired objects during listing.

Describe alternatives you've considered

Synchronize tree and object services in the node, which is not quite possible right now.

## Is your feature request related to a problem? Please describe. S3 Gateway provides a possibility to set an expiration for the objects. However the tree service and object service are not synced in the storage node. So expired object will not be removed from tree service, which may lead to unexpected listing results. ## Describe the solution you'd like Store expiration epoch of the object in the tree and ignore expired objects during listing. ## Describe alternatives you've considered Synchronize tree and object services in the node, which is not quite possible right now.

alexvanin commented

2023-06-27 06:28:36 +00:00

Poster

Owner

So expired object will not be removed from tree service, which may lead to unexpected listing results.

Maybe it shouldn't because gateway sends HEAD requests during listing. But anyway we are going to have dead records in the tree, which isn't a great thing.

Gateway can remove expired objects from the tree every time it sees it.

> So expired object will not be removed from tree service, which may lead to unexpected listing results. Maybe it shouldn't because gateway sends HEAD requests during listing. But anyway we are going to have dead records in the tree, which isn't a great thing. Gateway can remove expired objects from the tree every time it sees it.

dstepanov-yadro commented

2023-06-28 06:36:28 +00:00

It is necessary to provide for such a scenario: there is an object with a expireAt=10, it is blocked by a lock with expireAt=20. Then the object will actually be deleted at epoch 20, not 10.
There may also be several locks with different expireAt.

It is necessary to provide for such a scenario: there is an object with a expireAt=10, it is blocked by a lock with expireAt=20. Then the object will actually be deleted at epoch 20, not 10. There may also be several locks with different expireAt.

alexvanin referenced this issue

2023-07-07 09:15:56 +00:00

Improve object listing speed #165

alexvanin commented

2024-01-29 06:48:16 +00:00

Poster

Owner

@dkirillov please look at this in the case of #294.

I don't think S3 Gateway works with neither lock or expiration objects right now. If it does, then consider adding some more data to the tree nodes.

@dkirillov please look at this in the case of #294. I don't think S3 Gateway works with neither lock or expiration objects right now. If it does, then consider adding some more data to the tree nodes.

dkirillov was assigned by alexvanin

2024-01-29 06:48:24 +00:00

dkirillov commented

2024-01-29 08:40:22 +00:00

Collaborator

S3 Gateway doesn't work with expiration of user-uploaded object rigth now (we have #42 for that).
For locks the situation is little different:

we can add lock for object (but we cannot manually delete this: therefore for retention lock we have to wait until lock is expired but if legal hold is set then the object is locked forever)

S3 Gateway doesn't work with expiration of user-uploaded object rigth now (we have #42 for that). For locks the situation is little different: * we can add lock for object (but we cannot manually delete this: therefore for retention lock we have to wait until lock is expired but if legal hold is set then the object is locked forever)