Do not send 500 when storage returns 'object already removed' #626

New issue

Closed

opened 2025-02-07 07:57:44 +00:00 by alexvanin · 1 comment

alexvanin commented

2025-02-07 07:57:44 +00:00

Owner

Expected Behavior

When storage returns 'object already removed', gateway returns 404 NoSuchKey instead of 500.

Current Behavior

Gateway returns 500 Internal Error

debug   get tree node   {"request_id": "1164643e-5328-462d-b620-3a77afd9b0da", "cid": "G55C8HXUeyGvgQNpGnH8Mhv2Kx49KsMv8f6RhWEH7HUW", "oid": "B6PBpNxbV17Har8PYnNdHNZKw7uAQTzCoRS9CD2atbg6"}
 debug   policy request  {"request_id": "1164643e-5328-462d-b620-3a77afd9b0da", "action": "s3:GetObject", "resource": "arn:aws:s3:::fix1/foo", "request properties": {"AccessBox-Attribute/FilePath":"1738911098_access.box","AccessBox-Attribute/IAM-MFA":"false","AccessBox-Attribute/Timestamp":"1738911098","AccessBox-Attribute/__SYSTEM__EXPIRATION_EPOCH":"10801","Owner":"NWeByJPgNC97F83hTUnSbnZSBKaFvk5HNw","frostfs:sourceIP":"172.33.0.1","frostfsid:groupID":""}, "resource properties": {}}
 error   request failed  {"request_id": "1164643e-5328-462d-b620-3a77afd9b0da", "method": "GetObject", "bucket": "fix1", "object": "foo", "description": "could not find object", "user": "NWeByJPgNC97F83hTUnSbnZSBKaFvk5HNw", "error": "read object header via connection pool: head object via client localhost:8080: read object header via client: status: code = 2052 message = object already removed", "status": 500}

Possible Solution

As far as I see, gateway gets this error in GetExtendedObjectInfo while trying to head the object by using headVersion and headLastVersionIfNotDeleted methods. Those methods may parse object already removed error as well as they parse not found error.

Steps to Reproduce (for bugs)

Upload an S3 object
Remove it using frostfs-cli
Try to get S3 object from s3 gateway

Context

Regression

Your Environment

s3-gw v0.32.7

## Expected Behavior When storage returns 'object already removed', gateway returns 404 NoSuchKey instead of 500. ## Current Behavior Gateway returns 500 Internal Error ``` debug get tree node {"request_id": "1164643e-5328-462d-b620-3a77afd9b0da", "cid": "G55C8HXUeyGvgQNpGnH8Mhv2Kx49KsMv8f6RhWEH7HUW", "oid": "B6PBpNxbV17Har8PYnNdHNZKw7uAQTzCoRS9CD2atbg6"} debug policy request {"request_id": "1164643e-5328-462d-b620-3a77afd9b0da", "action": "s3:GetObject", "resource": "arn:aws:s3:::fix1/foo", "request properties": {"AccessBox-Attribute/FilePath":"1738911098_access.box","AccessBox-Attribute/IAM-MFA":"false","AccessBox-Attribute/Timestamp":"1738911098","AccessBox-Attribute/__SYSTEM__EXPIRATION_EPOCH":"10801","Owner":"NWeByJPgNC97F83hTUnSbnZSBKaFvk5HNw","frostfs:sourceIP":"172.33.0.1","frostfsid:groupID":""}, "resource properties": {}} error request failed {"request_id": "1164643e-5328-462d-b620-3a77afd9b0da", "method": "GetObject", "bucket": "fix1", "object": "foo", "description": "could not find object", "user": "NWeByJPgNC97F83hTUnSbnZSBKaFvk5HNw", "error": "read object header via connection pool: head object via client localhost:8080: read object header via client: status: code = 2052 message = object already removed", "status": 500} ``` ## Possible Solution As far as I see, gateway gets this error in `GetExtendedObjectInfo` while trying to head the object by using `headVersion` and `headLastVersionIfNotDeleted` methods. Those methods may parse object `already removed` error as well as they parse `not found` error. ## Steps to Reproduce (for bugs) 1. Upload an S3 object 2. Remove it using frostfs-cli 3. Try to get S3 object from s3 gateway ## Context ## Regression No ## Your Environment s3-gw v0.32.7