System behaviour when evacuating REP 1 objects #115

New issue

Closed

opened 2023-03-08 23:34:43 +00:00 by snegurochka · 2 comments

snegurochka commented

2023-03-08 23:34:43 +00:00

Member

Original issue: https://github.com/nspcc-dev/neofs-node/issues/1812

We want to behave correctly when evacuating objects stored only on a single node.
Here are 2 situations:

Move node to maintenance -> evacuate objects
Evacuate objects -> move node to maintenance

In (1) the objects could be unavailable between the first and the second step.
In (2) the objects could become unavailable if policer on another node removes them right after the evacuation.

So we need to take special care here. The proposal is following:

Allow node to answer some requests in maintenance mode if possible.
Change its state in the netmap.
Make sure policer takes netmap into account even if a node answers requests: if an object is stored on a node that is under maintenance in a single replica, make sure it is replicated.

Original issue: https://github.com/nspcc-dev/neofs-node/issues/1812 We want to behave correctly when evacuating objects stored only on a single node. Here are 2 situations: 1. Move node to maintenance -> evacuate objects 2. Evacuate objects -> move node to maintenance In (1) the objects could be unavailable between the first and the second step. In (2) the objects could become unavailable if policer on another node removes them right after the evacuation. So we need to take special care here. The proposal is following: 1. Allow node to answer some requests in maintenance mode if possible. 2. Change its state in the netmap. 3. Make sure policer takes netmap into account even if a node answers requests: if an object is stored on a node that is under maintenance in a single replica, make sure it is replicated.

fyrchik commented

2023-04-07 17:23:13 +00:00

Owner

Decided to leave it as is for now, REP 1 is already a non-recommended policy.

fyrchik closed this issue

2023-04-07 17:23:13 +00:00

dstepanov-yadro commented

2024-02-01 09:28:26 +00:00

Member

Scenario:

I want to transfer all objects from node 1 to other nodes. The objects are saved with the REP 1 policy.
I switch all the shards on node 1 to read-only mode and start the evacuation. Evacuation knows that it cannot transfer data to any of the available shards and begins to transfer it to other nodes.
On other nodes, the policer is triggered, it takes an object that was moved by evacuation, checks that the object is already on node 1, and deletes this object from itself.
The evacuation is coming to an end, I am fully confident that everything is OK, I am switching off the node 1.
Node 1 switched off, policer deleted object on the other nodes, object lost. Profit!

Scenario: 1. I want to transfer all objects from node 1 to other nodes. The objects are saved with the REP 1 policy. 2. I switch all the shards on node 1 to read-only mode and start the evacuation. Evacuation knows that it cannot transfer data to any of the available shards and begins to transfer it to other nodes. 3. On other nodes, the policer is triggered, it takes an object that was moved by evacuation, checks that the object is already on node 1, and deletes this object from itself. 4. The evacuation is coming to an end, I am fully confident that everything is OK, I am switching off the node 1. 5. Node 1 switched off, policer deleted object on the other nodes, object lost. Profit!