[NEW TEST]: Request forwarding (without data duplication) #15

New issue

Open

opened 2023-03-09 09:09:18 +00:00 by JuliaKovshova · 0 comments

JuliaKovshova commented

2023-03-09 09:09:18 +00:00

Member

original issue: https://github.com/nspcc-dev/neofs-testcases/issues/448

Context
A bug related to request forwarding was found and fixed. Request forwarding is an optimization: when a node is not able to process GET locally (obj not found or any other non-logical error) it just wraps the original request and sends it to another node (that probably could handle that request better) AND does not create a new request from scratch.

Solved problem
A node has an original stream -- the aim of the operation and a list of the container nodes to which it could try to forward the request. Any object part it receives from a container node is sent to the original requester via the original stream immediately. If any error appears in forwarding, a node repeats the same forwarding process to another node from the beginning and resends all the object parts it has already sent. That duplication was solved in the attached PR.

Test that could fix and check that behavior
Create a container that does not include all the nodes in the network (e.g. ... CBF 1 SELECT 2 FROM * if there are 4 nodes);
Put an object huge enough to guarantee that getting that object will take measurable time (to be sure that the test is possible to shutdown a node after GET started but before GET stopped), it could be big in NeoFS terms;
See what nodes store the object; could be done via neofs-cli object head ... --ttl 1 or via neofs-cli container nodes (if REP is equal to CBF * SELECT all the nodes from the output should store the obj); alternatively, you can just make your container static via container policy and know what exactly nodes store the object;
Send GET/GETRANGE request to a node that does not store the object, that node is expected to forward such requests and to stream the responses;
Make some nodes that store the object stop, it will break streaming;
A node-forwarder should repeat the request to another container node and continue streaming only the parts that have not been sent yet, in other words, GET and GETRANGE should finish correctly, and the res object must contain a correct payload.

original issue: https://github.com/nspcc-dev/neofs-testcases/issues/448 Context A bug related to request forwarding was found and fixed. Request forwarding is an optimization: when a node is not able to process GET locally (obj not found or any other non-logical error) it just wraps the original request and sends it to another node (that probably could handle that request better) AND does not create a new request from scratch. Solved problem A node has an original stream -- the aim of the operation and a list of the container nodes to which it could try to forward the request. Any object part it receives from a container node is sent to the original requester via the original stream immediately. If any error appears in forwarding, a node repeats the same forwarding process to another node from the beginning and resends all the object parts it has already sent. That duplication was solved in the attached PR. Test that could fix and check that behavior Create a container that does not include all the nodes in the network (e.g. ... CBF 1 SELECT 2 FROM * if there are 4 nodes); Put an object huge enough to guarantee that getting that object will take measurable time (to be sure that the test is possible to shutdown a node after GET started but before GET stopped), it could be big in NeoFS terms; See what nodes store the object; could be done via neofs-cli object head ... --ttl 1 or via neofs-cli container nodes (if REP is equal to CBF * SELECT all the nodes from the output should store the obj); alternatively, you can just make your container static via container policy and know what exactly nodes store the object; Send GET/GETRANGE request to a node that does not store the object, that node is expected to forward such requests and to stream the responses; Make some nodes that store the object stop, it will break streaming; A node-forwarder should repeat the request to another container node and continue streaming only the parts that have not been sent yet, in other words, GET and GETRANGE should finish correctly, and the res object must contain a correct payload.