Object loss: (not so) redundant local copy removed on all machines #92
Labels
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No Milestone
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-node#92
Loading…
Reference in New Issue
There is no content yet.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?
Original issue: https://github.com/nspcc-dev/neofs-node/issues/2267
Expected Behavior
Object storage system should store objects. I mean, you put them and then get them back at any time until you delete them.
Current Behavior
So we have a four-node network with
Hw57cmN31gCrqyEyKL5km31TFYETzQa3qk8DNECk6a4H
container using this policy:An
AV3Z7kpn8hxnaWWB2QRZUF9x8YhnyuB5ntC5fX6PRhz1
object was uploaded into this container some (pretty long) time ago. It's stored on nodes 3 and 4 (there were some movements before the incident, but it's not relevant) until this happens:and
Nodes 3 and 4 (holding the object) decide to move it to 1 and 2 at around the same time. Both fail to do so for some reason (which is not really important, replication can fail for a number of reasons). Both then delete their local copies. Object is gone. Forever.
Possible Solution
Looks like something is wrong in the logic ensuring a proper number of copies exists before deleting local one.
Context
Yeah, it's T5 testnet.
Your Environment
Node version 0.34.0.
Need to verify that all changes from the neofs-node have been picked.