Make GC handle expired lockers/tombstones correctly while some shards are read-only #1445

New issue

Open

opened 2024-10-23 08:35:37 +00:00 by a-savchuk · 0 comments

a-savchuk commented

2024-10-23 08:35:37 +00:00

Member

The task is based on several bugs that have appeared recently. The following description concerns lockers and locks, but those considerations are also valid for tombstones.

Problem

Suppose a locked object and its locker are placed on separate shards. While handling expired lock objects, the GC deletes the expired locker but doesn't unlock the object if it's on a read-only shard. As the result, the original object is locked forever on that node.

Since the GC needs to access all shards when handling a expired locker, it uses callbacks passed from a storage engine. Then the GC accesses all shards separately, which leads to the problem described above.

Steps to reproduce

Lock an object such that its locker is placed on another shard
Set the shard with the object and lock to read-only
Wait for the locker to expire and be deleted by the GC
Set the shard with the object and lock to read-write
Try to delete the original object

Expected behavior

The object can be deleted in the future because it'll be eventually unlocked.

Current behavior

The object can't be deleted because it keeps being locked forever on that node.

Possible solutions

In addition to enforcing placement policies, make a policer unlock objects if there's no related locker
Make the GC to delete lockers only after unlocking all related objects. Here’s some context for this solution:
- Since the GC needs to access all shards when handling a expired locker, it uses callbacks passed from a storage engine
- Then the GC accesses all shards separately, which leads to the problem described above
- The idea of this solution is to change that behavior so that the GC first iterates through all shards to unlock objects. Afterward, it should delete lockers only for successfully unlocked objects and shouldn’t delete others, i.e. keep them for the next GC attempt

Please share your thoughts!

The task is based on several bugs that have appeared recently. The following description concerns lockers and locks, but those considerations are also valid for tombstones. ## Problem Suppose a locked object and its locker are placed on separate shards. While handling expired lock objects, the GC deletes the expired locker but doesn't unlock the object if it's on a read-only shard. As the result, the original object is locked forever on that node. Since the GC needs to access all shards when handling a expired locker, it uses callbacks passed from a storage engine. Then the GC accesses all shards separately, which leads to the problem described above. ![exp-lock-ok.png](/attachments/71ca5815-27ab-45f3-ba25-1aef9604a040) ![exp-lock-not-ok.png](/attachments/506107eb-2816-420a-9d52-e6048b7b4bb2) ## Steps to reproduce 1. Lock an object such that its locker is placed on another shard 2. Set the shard with the object and lock to read-only 3. Wait for the locker to expire and be deleted by the GC 4. Set the shard with the object and lock to read-write 5. Try to delete the original object ## Expected behavior The object can be deleted in the future because it'll be eventually unlocked. ## Current behavior The object can't be deleted because it keeps being locked forever on that node. ## Possible solutions - In addition to enforcing placement policies, make a policer unlock objects if there's no related locker - Make the GC to delete lockers only after unlocking all related objects. Here’s some context for this solution: - Since the GC needs to access all shards when handling a expired locker, it uses callbacks passed from a storage engine - Then the GC accesses all shards separately, which leads to the problem described above - The idea of this solution is to change that behavior so that the GC first iterates through all shards to unlock objects. Afterward, it should delete lockers only for successfully unlocked objects and shouldn’t delete others, i.e. keep them for the next GC attempt Please share your thoughts!

exp-lock-ok.png

48 KiB

exp-lock-not-ok.png

45 KiB

👍 1