GC cannot remove tombstones with error "object not found" #861

New issue

Closed

opened 2023-12-12 11:21:20 +00:00 by anikeev-yadro · 0 comments

anikeev-yadro commented

2023-12-12 11:21:20 +00:00

Member

Expected Behavior

GC should remove tombstones

Current Behavior

GC cannot remove tombstones with error "object not found"

Steps to Reproduce (for bugs)

Run failover test with reboot node shile load is running
After test completed see a lot of errors in log

Dec 12 09:56:01 artrock-node1 frostfs-node[1555873]: 2023-12-12T09:56:01.117Z        debug        shard/delete.go:101        can't remove object from blobStor        {"shard_id": "Djk3ULuYXdXa18qahpgcgo", "object_address": "F6bViADEF494jTEQuShBTtZLFq1VkkCfSfBfpjrxKbhY/2D943XrW2aAdFCCsygrs1xQToJKcJKtBzfj6WMvfpfs5", "error": "status: code = 2049 message = object not found"}

CG metric with status="failed_to_delete" is too big

# HELP frostfs_node_garbage_collector_deleted_objects_total Total count of objects GC deleted or failed to delete from disk
# TYPE frostfs_node_garbage_collector_deleted_objects_total counter
frostfs_node_garbage_collector_deleted_objects_total{shard_id="9y9GaCv7DfkaMSPQiczNFB",status="deleted"} 9579
frostfs_node_garbage_collector_deleted_objects_total{shard_id="9y9GaCv7DfkaMSPQiczNFB",status="failed_to_delete"} 147054

Regression

Version

FrostFS Storage node
Version: v0.37.0-rc.1-27-gc056bbb8
GoVersion: go1.20.5

Your Environment

HW
4 nodes

## Expected Behavior GC should remove tombstones ## Current Behavior GC cannot remove tombstones with error "object not found" ## Steps to Reproduce (for bugs) 1. Run failover test with reboot node shile load is running 2. After test completed see a lot of errors in log ``` Dec 12 09:56:01 artrock-node1 frostfs-node[1555873]: 2023-12-12T09:56:01.117Z debug shard/delete.go:101 can't remove object from blobStor {"shard_id": "Djk3ULuYXdXa18qahpgcgo", "object_address": "F6bViADEF494jTEQuShBTtZLFq1VkkCfSfBfpjrxKbhY/2D943XrW2aAdFCCsygrs1xQToJKcJKtBzfj6WMvfpfs5", "error": "status: code = 2049 message = object not found"} ``` 3. CG metric with `status="failed_to_delete"` is too big ``` # HELP frostfs_node_garbage_collector_deleted_objects_total Total count of objects GC deleted or failed to delete from disk # TYPE frostfs_node_garbage_collector_deleted_objects_total counter frostfs_node_garbage_collector_deleted_objects_total{shard_id="9y9GaCv7DfkaMSPQiczNFB",status="deleted"} 9579 frostfs_node_garbage_collector_deleted_objects_total{shard_id="9y9GaCv7DfkaMSPQiczNFB",status="failed_to_delete"} 147054 ``` ## Regression No ## Version ``` FrostFS Storage node Version: v0.37.0-rc.1-27-gc056bbb8 GoVersion: go1.20.5 ``` ## Your Environment HW 4 nodes