[#57] api: Clean up List and Name caches when object is missing in Tree service #71

ironbee · 2023-03-30T15:38:39Z

ironbee commented

2023-03-30 15:38:39 +00:00

No description provided.

ironbee requested review from fyrchik 2023-03-30 15:39:21 +00:00

ironbee requested review from alexvanin 2023-03-30 15:39:21 +00:00

ironbee requested review from dkirillov 2023-03-30 15:39:21 +00:00

dkirillov reviewed 2023-03-31 06:26:29 +00:00

api/layer/layer.go Outdated

					
				@ -549,6 +549,10 @@ func (n *layer) deleteObject(ctx context.Context, bkt *data.BucketInfo, settings

					if len(obj.VersionID) != 0 || settings.Unversioned() {

						var nodeVersion *data.NodeVersion

						if nodeVersion, obj.Error = n.getNodeVersionToDelete(ctx, bkt, obj); obj.Error != nil {

							if errors.IsS3Error(obj.Error, errors.ErrNoSuchKey) {

dkirillov commented

2023-03-31 06:26:29 +00:00

Maybe we can move this to dismissNotFoundError?
And make it

func (n *layer) handleNotFoundError(bkt *data.BucketInfo, obj *VersionedObject) error

Maybe we can move this to `dismissNotFoundError`? And make it ``` func (n *layer) handleNotFoundError(bkt *data.BucketInfo, obj *VersionedObject) error ```

dkirillov marked this conversation as resolved

ironbee force-pushed clean-up-cache-when-object-is-missing from 3f8a47c6d7 to 52094d2f2d

2023-03-31 15:17:50 +00:00

Compare

alexvanin approved these changes 2023-03-31 15:31:39 +00:00

dkirillov commented

2023-03-31 15:53:31 +00:00

If we want to handle the case when object is lost by node rather than by tree we have to add similar (to handleNotFoundError) function here and here

If we want to handle the case when object is lost by node rather than by tree we have to add similar (to `handleNotFoundError`) function [here](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/commit/6c68e217775c0b5c15a07f7c3ed61bfd48210868/api/layer/layer.go#L556) and [here](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/commit/6c68e217775c0b5c15a07f7c3ed61bfd48210868/api/layer/layer.go#L575)

ironbee commented

2023-03-31 15:57:04 +00:00

If we want to handle the case when object is lost by node rather than by tree we have to add similar (to handleNotFoundError) function here and here

I tested the scenario when blobstore is deleted rather than the pilorama. It went as expected: both object and bucket got deleted successfully.

> If we want to handle the case when object is lost by node rather than by tree we have to add similar (to `handleNotFoundError`) function [here](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/commit/6c68e217775c0b5c15a07f7c3ed61bfd48210868/api/layer/layer.go#L556) and [here](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/commit/6c68e217775c0b5c15a07f7c3ed61bfd48210868/api/layer/layer.go#L575) > I tested the scenario when blobstore is deleted rather than the pilorama. It went as expected: both object and bucket got deleted successfully.

dkirillov commented

2023-03-31 16:19:17 +00:00

If I did everything right:


$ aws s3api --endpoint http://localhost:8084 put-object  --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 --key obj
{
    "ETag": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}


// in s3 log
// 2023-03-31T19:10:06.814+0300    debug   layer/object.go:252     put object      {"reqId": "628538b3-f199-45c6-8b27-2f3e88defd61", "bucket": "dkirillov", "cid": "9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5", "object": "obj", "oid": "4KGGCPXQRqoAQuzy2A2kNRtmFpmxMXii54HmcqEGGbyV"}


$ aws s3api --endpoint http://localhost:8084 list-objects  --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 
{
    "Contents": [
        {
            "Key": "obj",
            "LastModified": "2023-03-31T16:10:06+00:00",
            "ETag": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
            "Size": 0,
            "Owner": {
                "DisplayName": "NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM",
                "ID": "NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM"
            }
        }
    ]
}


$ frostfs-cli -r "${DEV_ENV_NODE_1}" -w "${DEV_ENV_WALLET}" \
 object delete --cid 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 --oid 4KGGCPXQRqoAQuzy2A2kNRtmFpmxMXii54HmcqEGGbyV 


$ export AWS_MAX_ATTEMPTS=1 && aws s3api --endpoint http://localhost:8084 list-objects  --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5
{
    "Contents": [
        {
            "Key": "obj",
            "LastModified": "0001-01-01T00:00:00+00:00",
            "ETag": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
            "Size": 0,
            "Owner": {
                "DisplayName": "",
                "ID": ""
            }
        }
    ]
}


$ aws s3api --endpoint http://localhost:8084 delete-object  --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 --key obj

An error occurred (InternalError) when calling the DeleteObject operation (reached max retries: 0): We encountered an internal error, please try again.


// in s3 log
// 2023-03-31T19:12:03.428+0300    error   handler/util.go:29      call method     {"status": 500, "request_id": "1e1eb149-272c-4f84-93bb-b23953629b84", "method": "DeleteObject", "bucket": "9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5", "object": "obj", "description": "could not delete object", "error": "mark object removal via connection pool: failed to collect relatives: failed to get raw object header: read object header via client: status: code = 2052 message = object already removed"}




$ aws s3api --endpoint http://localhost:8084 list-objects  --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5
{
    "Contents": [
        {
            "Key": "obj",
            "LastModified": "0001-01-01T00:00:00+00:00",
            "ETag": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
            "Size": 0,
            "Owner": {
                "DisplayName": "",
                "ID": ""
            }
        }
    ]
}


$ aws s3api --endpoint http://localhost:8084 delete-bucket  --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5

An error occurred (BucketNotEmpty) when calling the DeleteBucket operation (reached max retries: 0): The bucket you tried to delete is not empty

s3-gw caches config:

cache:
  objects:
    lifetime: 10m
    size: 150
  list:
    lifetime: 10m
    size: 1000
  names:
    lifetime: 10m
    size: 1000
  accessbox:
    lifetime: 50m
    size: 10

If I did everything right: ``` $ aws s3api --endpoint http://localhost:8084 put-object --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 --key obj { "ETag": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" } // in s3 log // 2023-03-31T19:10:06.814+0300 debug layer/object.go:252 put object {"reqId": "628538b3-f199-45c6-8b27-2f3e88defd61", "bucket": "dkirillov", "cid": "9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5", "object": "obj", "oid": "4KGGCPXQRqoAQuzy2A2kNRtmFpmxMXii54HmcqEGGbyV"} $ aws s3api --endpoint http://localhost:8084 list-objects --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 { "Contents": [ { "Key": "obj", "LastModified": "2023-03-31T16:10:06+00:00", "ETag": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "Size": 0, "Owner": { "DisplayName": "NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM", "ID": "NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM" } } ] } $ frostfs-cli -r "${DEV_ENV_NODE_1}" -w "${DEV_ENV_WALLET}" \ object delete --cid 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 --oid 4KGGCPXQRqoAQuzy2A2kNRtmFpmxMXii54HmcqEGGbyV $ export AWS_MAX_ATTEMPTS=1 && aws s3api --endpoint http://localhost:8084 list-objects --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 { "Contents": [ { "Key": "obj", "LastModified": "0001-01-01T00:00:00+00:00", "ETag": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "Size": 0, "Owner": { "DisplayName": "", "ID": "" } } ] } $ aws s3api --endpoint http://localhost:8084 delete-object --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 --key obj An error occurred (InternalError) when calling the DeleteObject operation (reached max retries: 0): We encountered an internal error, please try again. // in s3 log // 2023-03-31T19:12:03.428+0300 error handler/util.go:29 call method {"status": 500, "request_id": "1e1eb149-272c-4f84-93bb-b23953629b84", "method": "DeleteObject", "bucket": "9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5", "object": "obj", "description": "could not delete object", "error": "mark object removal via connection pool: failed to collect relatives: failed to get raw object header: read object header via client: status: code = 2052 message = object already removed"} $ aws s3api --endpoint http://localhost:8084 list-objects --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 { "Contents": [ { "Key": "obj", "LastModified": "0001-01-01T00:00:00+00:00", "ETag": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "Size": 0, "Owner": { "DisplayName": "", "ID": "" } } ] } $ aws s3api --endpoint http://localhost:8084 delete-bucket --bucket 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 An error occurred (BucketNotEmpty) when calling the DeleteBucket operation (reached max retries: 0): The bucket you tried to delete is not empty ``` s3-gw caches config: ``` cache: objects: lifetime: 10m size: 150 list: lifetime: 10m size: 1000 names: lifetime: 10m size: 1000 accessbox: lifetime: 50m size: 10 ```

👍 1

ironbee commented

2023-03-31 18:03:52 +00:00

$ frostfs-cli -r "${DEV_ENV_NODE_1}" -w "${DEV_ENV_WALLET}"
object delete --cid 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 --oid 4KGGCPXQRqoAQuzy2A2kNRtmFpmxMXii54HmcqEGGbyV

I didn't use frostfs-cli to delete the object. Used the aws cli instead.

Will check this scenario then.

> $ frostfs-cli -r "${DEV_ENV_NODE_1}" -w "${DEV_ENV_WALLET}" \ > object delete --cid 9m22awLNiQJsFntXopjcgNcduknVZmTYMVrA2Hkh1kL5 --oid 4KGGCPXQRqoAQuzy2A2kNRtmFpmxMXii54HmcqEGGbyV I didn't use frostfs-cli to delete the object. Used the aws cli instead. Will check this scenario then.

alexvanin referenced this pull request from TrueCloudLab/frostfs-node

2023-04-03 06:52:17 +00:00

Unexpected object delete response code #197

alexvanin commented

2023-04-03 06:54:29 +00:00

@ironbee @dkirillov By the way, API doesn't say that object.Delete operation may return 2052 object already removed code. So I made an issue on that: TrueCloudLab/frostfs-node#197

Regardless of the issue, let's process 2052 object already removed error as well in next S3 gateway release.

@ironbee @dkirillov By the way, API doesn't say that `object.Delete` operation may return `2052 object already removed` code. So I made an issue on that: https://git.frostfs.info/TrueCloudLab/frostfs-node/issues/197 Regardless of the issue, let's process `2052 object already removed` error as well in next S3 gateway release.

ironbee was assigned by alexvanin

2023-04-03 15:12:05 +00:00

dkirillov commented

2023-04-04 08:38:32 +00:00

Actually, to delete bucket when we not found object in object service, we have to update listing (to skip objects that exist only in tree service) or delete object from tree service.

Actually, to delete bucket when we not found object in object service, we have to update [listing](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/commit/01afa1cae4253d7dd375f1cf52d1f65e66d5ca06/api/layer/object.go#L591) (to skip objects that exist only in tree service) or delete object from tree service.

alexvanin approved these changes 2023-04-04 11:29:21 +00:00

alexvanin referenced this pull request

2023-04-04 12:28:58 +00:00

Deleting objects/bucket when object is missing or removed in metabase #78

dkirillov approved these changes 2023-04-04 13:59:32 +00:00

dkirillov commented

2023-04-04 13:59:56 +00:00

@ironbee Please, rebase this PR

👍 1

ironbee force-pushed clean-up-cache-when-object-is-missing from 52094d2f2d to fb99b26209

2023-04-04 14:15:15 +00:00

Compare