TrueCloudLab/frostfs-s3-gw
17
3
Fork 19
Code Issues 28 Pull requests 6 Projects Releases 18 Packages Activity Actions

Delete object through boto3 protocol returned error Object not found #57

New issue
Closed
opened 2023-03-14 11:50:06 +00:00 by anikeev-yadro · 14 comments
anikeev-yadro commented 2023-03-14 11:50:06 +00:00
Member
Copy link

Steps to reproduce:

  1. Put ob object into bucket using boto3:
COMMAND: 'S3 Put object result'
OUTPUT:
 {
    "ETag": "e9a6a6f0d35879b291858ab2ef576ec4c9fedb2943083913d550223a7c654474",
    "ResponseMetadata": {
        "HTTPHeaders": {
            "accept-ranges": "bytes",
            "content-length": "0",
            "date": "Tue, 14 Mar 2023 10:49:47 GMT",
            "etag": "e9a6a6f0d35879b291858ab2ef576ec4c9fedb2943083913d550223a7c654474",
            "server": "FrostFS-S3-GW/v0.0.1-78-g0af06c3b",
            "x-amz-request-id": "868be2a5-6a92-4d23-a0fd-6c41a1c7e141"
        },
        "HTTPStatusCode": 200,
        "HostId": "",
        "RequestId": "868be2a5-6a92-4d23-a0fd-6c41a1c7e141",
        "RetryAttempts": 0
    }
}
  1. Check that object exist in bucket
COMMAND: 'S3 List objects result'
OUTPUT:
 {'ResponseMetadata': {'RequestId': 'acc1cac0-2a44-4b56-a60c-e71e81b2faaf', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': 'acc1cac0-2a44-4b56-a60c-e71e81b2faaf', 'date': 'Tue, 14 Mar 2023 10:49:47 GMT', 'content-length': '624', 'content-type': 'text/xml; charset=utf-8'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Marker': '', 'Contents': [{'Key': '2cd4cf47-6e02-41cd-8194-a92d2ed00428', 'LastModified': datetime.datetime(2023, 3, 14, 10, 48, 58, tzinfo=tzlocal()), 'ETag': 'e9a6a6f0d35879b291858ab2ef576ec4c9fedb2943083913d550223a7c654474', 'Size': 1000, 'Owner': {'DisplayName': 'NcrzbL1UnY52YxQhPvZaDuh8V14UrGYAe3', 'ID': 'NcrzbL1UnY52YxQhPvZaDuh8V14UrGYAe3'}}], 'Name': '2f7a6bad-44d7-4e32-9e8c-beedde077ede', 'Prefix': '', 'MaxKeys': 1000, 'EncodingType': 'url'}
  1. Stop frostfs-storage service on all nodes
  2. Remove all nodes from map using following frostfs-adm --config /home/service/config.yaml morph remove-nodes command
  3. Start frostfs-storage service on all nodes
  4. Tick epoch and make sure that all nodes returned to map
  5. Make sure that object presented in bucket (only one object in bucket)
COMMAND: 'S3 List objects result'
OUTPUT:
 {'ResponseMetadata': {'RequestId': '6449ac4d-899f-4bb8-8721-3de7b719a811', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': '6449ac4d-899f-4bb8-8721-3de7b719a811', 'date': 'Tue, 14 Mar 2023 10:52:25 GMT', 'content-length': '556', 'content-type': 'text/xml; charset=utf-8'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Marker': '', 'Contents': [{'Key': '2cd4cf47-6e02-41cd-8194-a92d2ed00428', 'LastModified': datetime.datetime(1, 1, 1, 0, 0, tzinfo=tzlocal()), 'ETag': 'e9a6a6f0d35879b291858ab2ef576ec4c9fedb2943083913d550223a7c654474', 'Size': 1000, 'Owner': {'DisplayName': '', 'ID': ''}}], 'Name': '2f7a6bad-44d7-4e32-9e8c-beedde077ede', 'Prefix': '', 'MaxKeys': 1000, 'EncodingType': 'url'}
  1. Delete object returned error Object not found:
COMMAND: 'S3 Delete objects result'
OUTPUT:
 {
    "Errors": [
        {
            "Code": "BadRequest",
            "Key": "2cd4cf47-6e02-41cd-8194-a92d2ed00428",
            "Message": "mark object removal via connection pool: failed to collect relatives: failed to get raw object header: read object header via client: status: code = 2049 message = object not found"
        }
    ],
    "ResponseMetadata": {
        "HTTPHeaders": {
            "content-length": "400",
            "content-type": "text/xml; charset=utf-8",
            "date": "Tue, 14 Mar 2023 10:52:25 GMT",
            "x-amz-request-id": "bed61d19-c71f-4a46-8e54-b4fdc23e507a"
        },
        "HTTPStatusCode": 200,
        "HostId": "",
        "RequestId": "bed61d19-c71f-4a46-8e54-b4fdc23e507a",
        "RetryAttempts": 0
    }
}
  1. Then 'delete bucket' returned that bucket is not empty
COMMAND: 'S3 Delete bucket error'
OUTPUT:
 "An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty"

We need to understand why delete object returned error Object not found if bucket is not empty and contains only this object.

Versions

AWS
2.2.0

node
0.0.1-205-g9929dcf5

s3-gate
0.0.1-78-g0af06c3b

Your Environment

Server setup and configuration:
Virtual, 4 servers, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

## Steps to reproduce: 1. Put ob object into bucket using boto3: ``` COMMAND: 'S3 Put object result' OUTPUT: { "ETag": "e9a6a6f0d35879b291858ab2ef576ec4c9fedb2943083913d550223a7c654474", "ResponseMetadata": { "HTTPHeaders": { "accept-ranges": "bytes", "content-length": "0", "date": "Tue, 14 Mar 2023 10:49:47 GMT", "etag": "e9a6a6f0d35879b291858ab2ef576ec4c9fedb2943083913d550223a7c654474", "server": "FrostFS-S3-GW/v0.0.1-78-g0af06c3b", "x-amz-request-id": "868be2a5-6a92-4d23-a0fd-6c41a1c7e141" }, "HTTPStatusCode": 200, "HostId": "", "RequestId": "868be2a5-6a92-4d23-a0fd-6c41a1c7e141", "RetryAttempts": 0 } } ``` 2. Check that object exist in bucket ``` COMMAND: 'S3 List objects result' OUTPUT: {'ResponseMetadata': {'RequestId': 'acc1cac0-2a44-4b56-a60c-e71e81b2faaf', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': 'acc1cac0-2a44-4b56-a60c-e71e81b2faaf', 'date': 'Tue, 14 Mar 2023 10:49:47 GMT', 'content-length': '624', 'content-type': 'text/xml; charset=utf-8'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Marker': '', 'Contents': [{'Key': '2cd4cf47-6e02-41cd-8194-a92d2ed00428', 'LastModified': datetime.datetime(2023, 3, 14, 10, 48, 58, tzinfo=tzlocal()), 'ETag': 'e9a6a6f0d35879b291858ab2ef576ec4c9fedb2943083913d550223a7c654474', 'Size': 1000, 'Owner': {'DisplayName': 'NcrzbL1UnY52YxQhPvZaDuh8V14UrGYAe3', 'ID': 'NcrzbL1UnY52YxQhPvZaDuh8V14UrGYAe3'}}], 'Name': '2f7a6bad-44d7-4e32-9e8c-beedde077ede', 'Prefix': '', 'MaxKeys': 1000, 'EncodingType': 'url'} ``` 3. Stop frostfs-storage service on all nodes 4. Remove all nodes from map using following `frostfs-adm --config /home/service/config.yaml morph remove-nodes` command 5. Start frostfs-storage service on all nodes 6. Tick epoch and make sure that all nodes returned to map 7. Make sure that object presented in bucket (only one object in bucket) ``` COMMAND: 'S3 List objects result' OUTPUT: {'ResponseMetadata': {'RequestId': '6449ac4d-899f-4bb8-8721-3de7b719a811', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': '6449ac4d-899f-4bb8-8721-3de7b719a811', 'date': 'Tue, 14 Mar 2023 10:52:25 GMT', 'content-length': '556', 'content-type': 'text/xml; charset=utf-8'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Marker': '', 'Contents': [{'Key': '2cd4cf47-6e02-41cd-8194-a92d2ed00428', 'LastModified': datetime.datetime(1, 1, 1, 0, 0, tzinfo=tzlocal()), 'ETag': 'e9a6a6f0d35879b291858ab2ef576ec4c9fedb2943083913d550223a7c654474', 'Size': 1000, 'Owner': {'DisplayName': '', 'ID': ''}}], 'Name': '2f7a6bad-44d7-4e32-9e8c-beedde077ede', 'Prefix': '', 'MaxKeys': 1000, 'EncodingType': 'url'} ``` 8. Delete object returned error `Object not found`: ``` COMMAND: 'S3 Delete objects result' OUTPUT: { "Errors": [ { "Code": "BadRequest", "Key": "2cd4cf47-6e02-41cd-8194-a92d2ed00428", "Message": "mark object removal via connection pool: failed to collect relatives: failed to get raw object header: read object header via client: status: code = 2049 message = object not found" } ], "ResponseMetadata": { "HTTPHeaders": { "content-length": "400", "content-type": "text/xml; charset=utf-8", "date": "Tue, 14 Mar 2023 10:52:25 GMT", "x-amz-request-id": "bed61d19-c71f-4a46-8e54-b4fdc23e507a" }, "HTTPStatusCode": 200, "HostId": "", "RequestId": "bed61d19-c71f-4a46-8e54-b4fdc23e507a", "RetryAttempts": 0 } } ``` 9. Then 'delete bucket' returned that bucket is not empty ``` COMMAND: 'S3 Delete bucket error' OUTPUT: "An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty" ``` We need to understand why delete object returned error `Object not found` if bucket is not empty and contains only this object. ## Versions ``` AWS 2.2.0 node 0.0.1-205-g9929dcf5 s3-gate 0.0.1-78-g0af06c3b ``` ## Your Environment Server setup and configuration: Virtual, 4 servers, 4 SN, 4 http qw, 4 s3 gw Operating System and version (uname -a): linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux
nodes_logs_86cabe18ba9fd5c.zip
812 KiB
test_log_a23c09278d6dca1.txt
193 KiB
realloc added the
P1
 label 2023-03-14 11:55:00 +00:00
dkirillov was assigned by alexvanin 2023-03-14 16:30:07 +00:00
dkirillov commented 2023-03-15 08:48:27 +00:00
Member
Copy link

After such steps something goes wrong with tree service. It starts return not found: rpc error: code = Unknown desc = tree not found error.

So bucket is considered as empty. What you can see after list-objects command is cached values.

It seems this problem in node rather than in gate

/cc @alexvanin @fyrchik

After such steps something goes wrong with tree service. It starts return `not found: rpc error: code = Unknown desc = tree not found` error. So bucket is considered as empty. What you can see after `list-objects` command is cached values. It seems this problem in node rather than in gate /cc @alexvanin @fyrchik
dkirillov commented 2023-03-15 08:57:15 +00:00
Member
Copy link

It seems this has been already fixed in TrueCloudLab/frostfs-node#129

@anikeev-yadro

It seems this has been already fixed in https://git.frostfs.info/TrueCloudLab/frostfs-node/pulls/129 @anikeev-yadro
anikeev-yadro commented 2023-03-15 09:22:48 +00:00
Author
Member
Copy link

@dkirillov OK, If bucket is considered as empty, we should return empty list-objects and allow to delete empty bucket.
Maybe this is bug to cache?

@dkirillov OK, If bucket is considered as empty, we should return empty `list-objects` and allow to delete empty bucket. Maybe this is bug to cache?
dkirillov commented 2023-03-15 11:28:45 +00:00
Member
Copy link

@dkirillov OK, If bucket is considered as empty, we should return empty list-objects and allow to delete empty bucket.
Maybe this is bug to cache?

Sorry, I meant container (not bucket)

> @dkirillov OK, If bucket is considered as empty, we should return empty `list-objects` and allow to delete empty bucket. > Maybe this is bug to cache? Sorry, I meant container (not bucket)
alexvanin added this to the v0.27.0 milestone 2023-03-16 06:56:47 +00:00
dkirillov commented 2023-03-16 13:41:22 +00:00
Member
Copy link

I've just noticed that error in description from the node object service rather than tree service. So it seems that we loss object in container but in the tree service they exist.

I've just noticed that error in description from the node object service rather than tree service. So it seems that we loss object in container but in the tree service they exist.
dkirillov was unassigned by alexvanin 2023-03-17 15:20:23 +00:00
ironbee was assigned by alexvanin 2023-03-17 15:20:24 +00:00
dkirillov commented 2023-03-17 15:53:36 +00:00
Member
Copy link

It seems we need help @fyrchik @carpawell to know what happened with node after restart.

We also lose the access box object (probably all objects). But we can make request to s3-gw because of cache

It seems we need help @fyrchik @carpawell to know what happened with node after restart. We also lose the access box object (probably all objects). But we can make request to s3-gw because of cache
alexvanin commented 2023-04-04 14:58:34 +00:00
Owner
Copy link

In this issue we close the case with tree service miss + blobovnicza and fstree unavailability. However there are more things to process in #78 when metabase returns object not found and object already deleted. We need to discuss it there additionally.
/cc @anikeev-yadro

In this issue we close the case with tree service miss + blobovnicza and fstree unavailability. However there are more things to process in #78 when metabase returns `object not found` and `object already deleted`. We need to discuss it there additionally. /cc @anikeev-yadro
anikeev-yadro commented 2023-04-11 07:35:24 +00:00
Author
Member
Copy link

Reproduced on the following case:
1.Create bucket
2.Put object into bucket
3.Check that bucket contains the object (list object)
4.Stop frostfs-storage service on all nodes

root@aanikeev-node4:~# systemctl stop frostfs-storage

5.Delete all data on all nodes

root@aanikeev-node4:~# rm -rf /srv/frostfs/data*

6.Start frostfs-storage service on all nodes

root@aanikeev-node4:~# systemctl start frostfs-storage

7.Delete object

anikeev@NB-1670:~/neofs$ aws s3api delete-object --key new-object --bucket b-test-lock-6 --endpoint-url http://172.26.160.28:8084
{
    "DeleteMarker": true,
    "VersionId": "2miMkbvybPxGyxJuZo6CM7CocWof26HptTbvi2L6Mc4S"
}

8.Delete bucket

anikeev@NB-1670:~/neofs$ aws s3api delete-bucket --bucket b-test-lock-6 --endpoint-url http://172.26.160.28:8084
An error occurred (BucketNotEmpty) when calling the DeleteBucket operation (reached max retries: 0): The bucket you tried to delete is not empty

Version

FrostFS S3 Gateway
Version: v0.0.1-94-gfb99b262
GoVersion: go1.18.4
Reproduced on the following case: 1.Create bucket 2.Put object into bucket 3.Check that bucket contains the object (list object) 4.Stop `frostfs-storage` service on all nodes ``` root@aanikeev-node4:~# systemctl stop frostfs-storage ``` 5.Delete all data on all nodes ``` root@aanikeev-node4:~# rm -rf /srv/frostfs/data* ``` 6.Start `frostfs-storage` service on all nodes ``` root@aanikeev-node4:~# systemctl start frostfs-storage ``` 7.Delete object ``` anikeev@NB-1670:~/neofs$ aws s3api delete-object --key new-object --bucket b-test-lock-6 --endpoint-url http://172.26.160.28:8084 { "DeleteMarker": true, "VersionId": "2miMkbvybPxGyxJuZo6CM7CocWof26HptTbvi2L6Mc4S" } ``` 8.Delete bucket ``` anikeev@NB-1670:~/neofs$ aws s3api delete-bucket --bucket b-test-lock-6 --endpoint-url http://172.26.160.28:8084 An error occurred (BucketNotEmpty) when calling the DeleteBucket operation (reached max retries: 0): The bucket you tried to delete is not empty ``` ## Version ``` FrostFS S3 Gateway Version: v0.0.1-94-gfb99b262 GoVersion: go1.18.4 ```
dkirillov commented 2023-04-11 07:49:02 +00:00
Member
Copy link

7.Delete object
anikeev@NB-1670:~/neofs$ aws s3api delete-object --key new-object --bucket b-test-lock-6 --endpoint-url http://172.26.160.28:8084
{
"DeleteMarker": true,
"VersionId": "2miMkbvybPxGyxJuZo6CM7CocWof26HptTbvi2L6Mc4S"
}

It seems you are trying to delete object in versioned bucket. That's why you cannot delete bucket until you delete all versions/deletem markers https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html

> 7.Delete object anikeev@NB-1670:~/neofs$ aws s3api delete-object --key new-object --bucket b-test-lock-6 --endpoint-url http://172.26.160.28:8084 { "DeleteMarker": true, "VersionId": "2miMkbvybPxGyxJuZo6CM7CocWof26HptTbvi2L6Mc4S" } It seems you are trying to delete object in versioned bucket. That's why you cannot delete bucket until you delete all versions/deletem markers https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html
anikeev-yadro commented 2023-04-11 07:55:42 +00:00
Author
Member
Copy link

I agree, but main point is: Why the system allow to delete object and create delete marker even object deleted physically from disk?

I agree, but main point is: Why the system allow to delete object and create delete marker even object deleted physically from disk?
dkirillov commented 2023-04-11 10:07:16 +00:00
Member
Copy link

In current scenario we still have some object info in the tree service as I understand, so we don't actually know that object is deleted physically.

But we definitely should revise working with delete markers. #83

In current scenario we still have some object info in the tree service as I understand, so we don't actually know that object is deleted physically. But we definitely should revise working with delete markers. https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/issues/83
ironbee commented 2023-04-13 09:28:26 +00:00
Contributor
Copy link

Scenario 1: Bucket with versioning enabled, the object storage is lost but the tree remains.
If we try deleting an object in the bucket without specifying the version then a delete market gets created. We need to delete it along with the other versions of target object in order to delete the bucket. The process can be seen in S3 gateway log.

Scenario 1: Bucket with versioning enabled, the object storage is lost but the tree remains. If we try deleting an object in the bucket without specifying the version then a delete market gets created. We need to delete it along with the other versions of target object in order to delete the bucket. The process can be seen in S3 gateway log.
01_object_lost_s3cli_log.png
319 KiB
01_object_lost_s3gw_log.png
689 KiB
01_object_lost_s3cli_log.png 01_object_lost_s3gw_log.png
ironbee commented 2023-04-13 09:30:19 +00:00
Contributor
Copy link

Scenario 2: Bucket with versioning enabled, the tree is lost but the object storage remains.
Since the information on target object is not available we consider it as non-existing, thus the bucket can be deleted is needed.

Scenario 2: Bucket with versioning enabled, the tree is lost but the object storage remains. Since the information on target object is not available we consider it as non-existing, thus the bucket can be deleted is needed.
02_tree_lost_s3cli_log.png
188 KiB
02_tree_lost_s3gw_log.png
537 KiB
02_tree_lost_s3cli_log.png 02_tree_lost_s3gw_log.png
alexvanin commented 2023-04-14 13:21:24 +00:00
Owner
Copy link

Seems like it works as expected with versioned bucket as well.

Seems like it works as expected with versioned bucket as well.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
support/v0.30
support/v0.29
support/v0.28
support/v0.27
v0.31.0
v0.31.0-rc.6
v0.31.0-rc.5
v0.31.0-rc.4
v0.30.8
v0.31.0-rc.3
v0.30.7
v0.31.0-rc.2
v0.31.0-rc.1
v0.30.6
v0.30.5
v0.30.4
v0.30.3
v0.30.2
v0.30.1
v0.30.0
v0.29.3
v0.30.0-rc.5
v0.29.2
v0.30.0-rc.4
v0.30.0-rc.3
v0.30.0-rc.2
v0.29.1
v0.30.0-rc.1
v0.29.0
v0.28.2
v0.29.0-rc.11
v0.29.0-rc.10
v0.29.0-rc.9
v0.29.0-rc.8
v0.29.0-rc.7
v0.29.0-rc.6
v0.29.0-rc.5
v0.29.0-rc.4
v0.29.0-rc.3
v0.29.0-rc.2
v0.29.0-rc.1
v0.28.1
v0.28.0
v0.28.0-rc.1
v0.27.0
v0.27.0-rc.2
v0.27.0-rc.1
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
v0.27.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#57
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?