Improve object listing speed

alexvanin commented

2023-07-07 09:15:56 +00:00

Owner

Object listing produces HEAD requests for every object gateway finds in the tree service. This makes listing operation kinda slow but allows to exclude some objects from listing, e.g. expired or manually removed objects.

Consider some ways to improve searching speed:

Replace HEAD requests by expanding data stored in tree service (e.g. expiration epoch, see #141)
Make more parallel requests
Something else

Object listing produces `HEAD` requests for every object gateway finds in the tree service. This makes listing operation kinda slow but allows to exclude some objects from listing, e.g. expired or manually removed objects. Consider some ways to improve searching speed: 1) Replace HEAD requests by expanding data stored in tree service (e.g. expiration epoch, see #141) 2) Make more parallel requests 3) Something else

alexvanin added the

discussion

label 2023-07-07 09:15:56 +00:00

dkirillov commented

2023-07-07 09:33:52 +00:00

Collaborator

As I remember we wanted stream nodes from tree to be able paginate list results. But we need ordered GetSubTree first TrueCloudLab/frostfs-node#335

As I remember we wanted stream nodes from tree to be able paginate list results. But we need ordered GetSubTree first https://git.frostfs.info/TrueCloudLab/frostfs-node/issues/335

dkirillov was assigned by alexvanin

2023-09-07 12:09:08 +00:00

alexvanin added this to the v0.29.0 milestone 2023-09-08 10:31:27 +00:00

dkirillov referenced a pull request that will close this issue

2024-01-22 12:26:20 +00:00

feature/165-speed_up_listing #294

alexvanin commented

2024-01-26 08:31:20 +00:00

Poster

Owner

#294 implements solution that consists of two optimizations:

read and maintain gRPC stream of tree nodes instead of fetching the whole tree on every listing request,
keep all required data in the tree node therefore remove object.HEAD requests on every object in bucket.

While first optimization provides speed improvement, there are still many object.HEAD requests in large buckets with significant overhead. These requests had a purpose in the case when object service and tree service are unsynced: object may be unavailable while tree node is present.

During object removal, gateway sends object service request first and then removes tree node. Previously we've decided that it is dangerous to ignore object service failures during object removal. So node remains in the tree and gateway removes object from listing output after failed object.HEAD request (#78). This way bucket can be cleaned and removed regardless of object service failures and we avoid unnecessary tree node removal.

In order to apply second optimization, we should dismiss earlier assumptions. As we discussed, this optimization leads to much better user experience and is more valuable than highly conservative approach of keeping meta-data (tree node) during object removal in case of object service failures.

After this fix:

on bucket list request, objects will be present in output regardless of status in object service,
on object delete request, objects will be removed from tree service regardless of object service failures.

/cc @realloc @a.bogatyrev @fyrchik

#294 implements solution that consists of two optimizations: 1. read and maintain gRPC stream of tree nodes instead of fetching the whole tree on every listing request, 2. keep all required data in the tree node therefore remove `object.HEAD` requests on every object in bucket. While first optimization provides speed improvement, there are still many `object.HEAD` requests in large buckets with significant overhead. These requests had a purpose in the case when object service and tree service are unsynced: object may be unavailable while tree node is present. During object removal, gateway sends object service request first and then removes tree node. Previously we've decided that it is dangerous to ignore object service failures during object removal. So node remains in the tree and gateway removes object from listing output after failed `object.HEAD` request (#78). This way bucket can be cleaned and removed regardless of object service failures and we avoid unnecessary tree node removal. In order to apply second optimization, we should dismiss earlier assumptions. As we discussed, this optimization leads to much better user experience and is more valuable than highly conservative approach of keeping meta-data (tree node) during object removal in case of object service failures. After this fix: - on bucket list request, objects will be present in output regardless of status in object service, - on object delete request, objects will be removed from tree service regardless of object service failures. /cc @realloc @a.bogatyrev @fyrchik

👍 1

dkirillov commented

2024-01-26 09:03:04 +00:00

Collaborator

objects will be removed from tree service regardless of object service failures.

Only for object already removed and object not found failures

> objects will be removed from tree service regardless of object service failures. Only for `object already removed` and `object not found` failures

fyrchik commented

2024-01-26 09:11:47 +00:00

Owner

In frostfs-cli there is a special case for LOCK objects: if this "supposedly missing" object turns out to be of LOCK type, will this be S3 protocol violation? I mean deleting such bucket.