2018-01-26 18:04:13 +00:00
|
|
|
package network
|
|
|
|
|
|
|
|
import (
|
network: rework broadcast logic
We have a number of queues for different purposes:
* regular broadcast queue
* direct p2p queue
* high-priority queue
And two basic egress scenarios:
* direct p2p messages (replies to requests in Server's handle* methods)
* broadcasted messages
Low priority broadcasted messages:
* transaction inventories
* block inventories
* notary inventories
* non-consensus extensibles
High-priority broadcasted messages:
* consensus extensibles
* getdata transaction requests from consensus process
* getaddr requests
P2P messages are a bit more complicated, most of the time they use p2p queue,
but extensible message requests/replies use HP queue.
Server's handle* code is run from Peer's handleIncoming, every peer has this
thread that handles incoming messages. When working with the peer it's
important to reply to requests and blocking this thread until we send (queue)
a reply is fine, if the peer is slow we just won't get anything new from
it. The queue used is irrelevant wrt this issue.
Broadcasted messages are radically different, we want them to be delivered to
many peers, but we don't care about specific ones. If it's delivered to 2/3 of
the peers we're fine, if it's delivered to more of them --- it's not an
issue. But doing this fairly is not an easy thing, current code tries performing
unblocked sends and if this doesn't yield enough results it then blocks (but
has a timeout, we can't wait indefinitely). But it does so in sequential
manner, once the peer is chosen the code will wait for it (and only it) until
timeout happens.
What can be done instead is an attempt to push the message to all of the peers
simultaneously (or close to that). If they all deliver --- OK, if some block
and wait then we can wait until _any_ of them pushes the message through (or
global timeout happens, we still can't wait forever). If we have enough
deliveries then we can cancel pending ones and it's again not an error if
these canceled threads still do their job.
This makes the system more dynamic and adds some substantial processing
overhead, but it's a networking code, any of this overhead is much lower than
the actual packet delivery time. It also allows to spread the load more
fairly, if there is any spare queue it'll get the packet and release the
broadcaster. On the next broadcast iteration another peer is more likely to be
chosen just because it didn't get a message previously (and had some time to
deliver already queued messages).
It works perfectly in tests, with optimal networking conditions we have much
better block times and TPS increases by 5-25%% depending on the scenario.
I'd go as far as to say that it fixes the original problem of #2678, because
in this particular scenario we have empty queues in ~100% of the cases and
this new logic will likely lead to 100% fan out in this case (cancelation just
won't happen fast enough). But when the load grows and there is some waiting
in the queue it will optimize out the slowest links.
2022-10-10 19:48:06 +00:00
|
|
|
"context"
|
2019-09-09 14:54:38 +00:00
|
|
|
"net"
|
|
|
|
|
2020-03-03 14:21:42 +00:00
|
|
|
"github.com/nspcc-dev/neo-go/pkg/network/payload"
|
2018-01-26 18:04:13 +00:00
|
|
|
)
|
|
|
|
|
2019-09-03 14:51:37 +00:00
|
|
|
// Peer represents a network node neo-go is connected to.
|
2018-01-31 08:27:08 +00:00
|
|
|
type Peer interface {
|
2019-11-06 07:55:21 +00:00
|
|
|
// RemoteAddr returns the remote address that we're connected to now.
|
|
|
|
RemoteAddr() net.Addr
|
|
|
|
// PeerAddr returns the remote address that should be used to establish
|
|
|
|
// a new connection to the node. It can differ from the RemoteAddr
|
2022-04-20 18:30:09 +00:00
|
|
|
// address in case the remote node is a client and its current
|
2019-11-06 07:55:21 +00:00
|
|
|
// connection port is different from the one the other node should use
|
2022-04-20 18:30:09 +00:00
|
|
|
// to connect to it. It's only valid after the handshake is completed.
|
|
|
|
// Before that, it returns the same address as RemoteAddr.
|
2019-11-06 07:55:21 +00:00
|
|
|
PeerAddr() net.Addr
|
2018-03-14 09:36:59 +00:00
|
|
|
Disconnect(error)
|
2020-01-16 18:16:31 +00:00
|
|
|
|
network: rework broadcast logic
We have a number of queues for different purposes:
* regular broadcast queue
* direct p2p queue
* high-priority queue
And two basic egress scenarios:
* direct p2p messages (replies to requests in Server's handle* methods)
* broadcasted messages
Low priority broadcasted messages:
* transaction inventories
* block inventories
* notary inventories
* non-consensus extensibles
High-priority broadcasted messages:
* consensus extensibles
* getdata transaction requests from consensus process
* getaddr requests
P2P messages are a bit more complicated, most of the time they use p2p queue,
but extensible message requests/replies use HP queue.
Server's handle* code is run from Peer's handleIncoming, every peer has this
thread that handles incoming messages. When working with the peer it's
important to reply to requests and blocking this thread until we send (queue)
a reply is fine, if the peer is slow we just won't get anything new from
it. The queue used is irrelevant wrt this issue.
Broadcasted messages are radically different, we want them to be delivered to
many peers, but we don't care about specific ones. If it's delivered to 2/3 of
the peers we're fine, if it's delivered to more of them --- it's not an
issue. But doing this fairly is not an easy thing, current code tries performing
unblocked sends and if this doesn't yield enough results it then blocks (but
has a timeout, we can't wait indefinitely). But it does so in sequential
manner, once the peer is chosen the code will wait for it (and only it) until
timeout happens.
What can be done instead is an attempt to push the message to all of the peers
simultaneously (or close to that). If they all deliver --- OK, if some block
and wait then we can wait until _any_ of them pushes the message through (or
global timeout happens, we still can't wait forever). If we have enough
deliveries then we can cancel pending ones and it's again not an error if
these canceled threads still do their job.
This makes the system more dynamic and adds some substantial processing
overhead, but it's a networking code, any of this overhead is much lower than
the actual packet delivery time. It also allows to spread the load more
fairly, if there is any spare queue it'll get the packet and release the
broadcaster. On the next broadcast iteration another peer is more likely to be
chosen just because it didn't get a message previously (and had some time to
deliver already queued messages).
It works perfectly in tests, with optimal networking conditions we have much
better block times and TPS increases by 5-25%% depending on the scenario.
I'd go as far as to say that it fixes the original problem of #2678, because
in this particular scenario we have empty queues in ~100% of the cases and
this new logic will likely lead to 100% fan out in this case (cancelation just
won't happen fast enough). But when the load grows and there is some waiting
in the queue it will optimize out the slowest links.
2022-10-10 19:48:06 +00:00
|
|
|
// BroadcastPacket is a context-bound packet enqueuer, it either puts the
|
|
|
|
// given packet into the queue or exits with errors if the context expires
|
|
|
|
// or peer disconnects. It accepts a slice of bytes that
|
2020-01-16 18:16:31 +00:00
|
|
|
// can be shared with other queues (so that message marshalling can be
|
network: rework broadcast logic
We have a number of queues for different purposes:
* regular broadcast queue
* direct p2p queue
* high-priority queue
And two basic egress scenarios:
* direct p2p messages (replies to requests in Server's handle* methods)
* broadcasted messages
Low priority broadcasted messages:
* transaction inventories
* block inventories
* notary inventories
* non-consensus extensibles
High-priority broadcasted messages:
* consensus extensibles
* getdata transaction requests from consensus process
* getaddr requests
P2P messages are a bit more complicated, most of the time they use p2p queue,
but extensible message requests/replies use HP queue.
Server's handle* code is run from Peer's handleIncoming, every peer has this
thread that handles incoming messages. When working with the peer it's
important to reply to requests and blocking this thread until we send (queue)
a reply is fine, if the peer is slow we just won't get anything new from
it. The queue used is irrelevant wrt this issue.
Broadcasted messages are radically different, we want them to be delivered to
many peers, but we don't care about specific ones. If it's delivered to 2/3 of
the peers we're fine, if it's delivered to more of them --- it's not an
issue. But doing this fairly is not an easy thing, current code tries performing
unblocked sends and if this doesn't yield enough results it then blocks (but
has a timeout, we can't wait indefinitely). But it does so in sequential
manner, once the peer is chosen the code will wait for it (and only it) until
timeout happens.
What can be done instead is an attempt to push the message to all of the peers
simultaneously (or close to that). If they all deliver --- OK, if some block
and wait then we can wait until _any_ of them pushes the message through (or
global timeout happens, we still can't wait forever). If we have enough
deliveries then we can cancel pending ones and it's again not an error if
these canceled threads still do their job.
This makes the system more dynamic and adds some substantial processing
overhead, but it's a networking code, any of this overhead is much lower than
the actual packet delivery time. It also allows to spread the load more
fairly, if there is any spare queue it'll get the packet and release the
broadcaster. On the next broadcast iteration another peer is more likely to be
chosen just because it didn't get a message previously (and had some time to
deliver already queued messages).
It works perfectly in tests, with optimal networking conditions we have much
better block times and TPS increases by 5-25%% depending on the scenario.
I'd go as far as to say that it fixes the original problem of #2678, because
in this particular scenario we have empty queues in ~100% of the cases and
this new logic will likely lead to 100% fan out in this case (cancelation just
won't happen fast enough). But when the load grows and there is some waiting
in the queue it will optimize out the slowest links.
2022-10-10 19:48:06 +00:00
|
|
|
// done once for all peers). It returns an error if the peer has not yet
|
2020-01-16 18:16:31 +00:00
|
|
|
// completed handshaking.
|
network: rework broadcast logic
We have a number of queues for different purposes:
* regular broadcast queue
* direct p2p queue
* high-priority queue
And two basic egress scenarios:
* direct p2p messages (replies to requests in Server's handle* methods)
* broadcasted messages
Low priority broadcasted messages:
* transaction inventories
* block inventories
* notary inventories
* non-consensus extensibles
High-priority broadcasted messages:
* consensus extensibles
* getdata transaction requests from consensus process
* getaddr requests
P2P messages are a bit more complicated, most of the time they use p2p queue,
but extensible message requests/replies use HP queue.
Server's handle* code is run from Peer's handleIncoming, every peer has this
thread that handles incoming messages. When working with the peer it's
important to reply to requests and blocking this thread until we send (queue)
a reply is fine, if the peer is slow we just won't get anything new from
it. The queue used is irrelevant wrt this issue.
Broadcasted messages are radically different, we want them to be delivered to
many peers, but we don't care about specific ones. If it's delivered to 2/3 of
the peers we're fine, if it's delivered to more of them --- it's not an
issue. But doing this fairly is not an easy thing, current code tries performing
unblocked sends and if this doesn't yield enough results it then blocks (but
has a timeout, we can't wait indefinitely). But it does so in sequential
manner, once the peer is chosen the code will wait for it (and only it) until
timeout happens.
What can be done instead is an attempt to push the message to all of the peers
simultaneously (or close to that). If they all deliver --- OK, if some block
and wait then we can wait until _any_ of them pushes the message through (or
global timeout happens, we still can't wait forever). If we have enough
deliveries then we can cancel pending ones and it's again not an error if
these canceled threads still do their job.
This makes the system more dynamic and adds some substantial processing
overhead, but it's a networking code, any of this overhead is much lower than
the actual packet delivery time. It also allows to spread the load more
fairly, if there is any spare queue it'll get the packet and release the
broadcaster. On the next broadcast iteration another peer is more likely to be
chosen just because it didn't get a message previously (and had some time to
deliver already queued messages).
It works perfectly in tests, with optimal networking conditions we have much
better block times and TPS increases by 5-25%% depending on the scenario.
I'd go as far as to say that it fixes the original problem of #2678, because
in this particular scenario we have empty queues in ~100% of the cases and
this new logic will likely lead to 100% fan out in this case (cancelation just
won't happen fast enough). But when the load grows and there is some waiting
in the queue it will optimize out the slowest links.
2022-10-10 19:48:06 +00:00
|
|
|
BroadcastPacket(context.Context, []byte) error
|
|
|
|
|
|
|
|
// BroadcastHPPacket is the same as BroadcastPacket, but uses a high-priority
|
|
|
|
// queue.
|
|
|
|
BroadcastHPPacket(context.Context, []byte) error
|
2020-01-16 18:16:31 +00:00
|
|
|
|
2022-10-12 12:39:20 +00:00
|
|
|
// EnqueueP2PMessage is a blocking packet enqueuer, it doesn't return until
|
|
|
|
// it puts the given message into the queue. It returns an error if the peer
|
|
|
|
// has not yet completed handshaking. This queue is intended to be used for
|
|
|
|
// unicast peer to peer communication that is more important than broadcasts
|
network: rework broadcast logic
We have a number of queues for different purposes:
* regular broadcast queue
* direct p2p queue
* high-priority queue
And two basic egress scenarios:
* direct p2p messages (replies to requests in Server's handle* methods)
* broadcasted messages
Low priority broadcasted messages:
* transaction inventories
* block inventories
* notary inventories
* non-consensus extensibles
High-priority broadcasted messages:
* consensus extensibles
* getdata transaction requests from consensus process
* getaddr requests
P2P messages are a bit more complicated, most of the time they use p2p queue,
but extensible message requests/replies use HP queue.
Server's handle* code is run from Peer's handleIncoming, every peer has this
thread that handles incoming messages. When working with the peer it's
important to reply to requests and blocking this thread until we send (queue)
a reply is fine, if the peer is slow we just won't get anything new from
it. The queue used is irrelevant wrt this issue.
Broadcasted messages are radically different, we want them to be delivered to
many peers, but we don't care about specific ones. If it's delivered to 2/3 of
the peers we're fine, if it's delivered to more of them --- it's not an
issue. But doing this fairly is not an easy thing, current code tries performing
unblocked sends and if this doesn't yield enough results it then blocks (but
has a timeout, we can't wait indefinitely). But it does so in sequential
manner, once the peer is chosen the code will wait for it (and only it) until
timeout happens.
What can be done instead is an attempt to push the message to all of the peers
simultaneously (or close to that). If they all deliver --- OK, if some block
and wait then we can wait until _any_ of them pushes the message through (or
global timeout happens, we still can't wait forever). If we have enough
deliveries then we can cancel pending ones and it's again not an error if
these canceled threads still do their job.
This makes the system more dynamic and adds some substantial processing
overhead, but it's a networking code, any of this overhead is much lower than
the actual packet delivery time. It also allows to spread the load more
fairly, if there is any spare queue it'll get the packet and release the
broadcaster. On the next broadcast iteration another peer is more likely to be
chosen just because it didn't get a message previously (and had some time to
deliver already queued messages).
It works perfectly in tests, with optimal networking conditions we have much
better block times and TPS increases by 5-25%% depending on the scenario.
I'd go as far as to say that it fixes the original problem of #2678, because
in this particular scenario we have empty queues in ~100% of the cases and
this new logic will likely lead to 100% fan out in this case (cancelation just
won't happen fast enough). But when the load grows and there is some waiting
in the queue it will optimize out the slowest links.
2022-10-10 19:48:06 +00:00
|
|
|
// (handled by BroadcastPacket) but less important than high-priority
|
2022-10-12 12:39:20 +00:00
|
|
|
// messages (handled by EnqueueHPMessage).
|
|
|
|
EnqueueP2PMessage(*Message) error
|
2022-10-21 07:49:44 +00:00
|
|
|
// EnqueueP2PPacket is similar to EnqueueP2PMessage, but accepts a slice of
|
|
|
|
// message(s) bytes.
|
|
|
|
EnqueueP2PPacket([]byte) error
|
2020-01-23 16:40:40 +00:00
|
|
|
|
2022-10-12 12:39:20 +00:00
|
|
|
// EnqueueHPMessage is similar to EnqueueP2PMessage, but uses a high-priority
|
2020-01-16 18:16:31 +00:00
|
|
|
// queue.
|
2022-10-12 12:39:20 +00:00
|
|
|
EnqueueHPMessage(*Message) error
|
2022-10-21 07:49:44 +00:00
|
|
|
// EnqueueHPPacket is similar to EnqueueHPMessage, but accepts a slice of
|
|
|
|
// message(s) bytes.
|
|
|
|
EnqueueHPPacket([]byte) error
|
2018-03-14 09:36:59 +00:00
|
|
|
Version() *payload.Version
|
2020-01-17 10:17:19 +00:00
|
|
|
LastBlockIndex() uint32
|
2019-09-13 12:43:22 +00:00
|
|
|
Handshaked() bool
|
2020-05-22 09:17:17 +00:00
|
|
|
IsFullNode() bool
|
2020-01-20 16:02:19 +00:00
|
|
|
|
2022-10-12 12:25:03 +00:00
|
|
|
// SetPingTimer adds an outgoing ping to the counter and sets a PingTimeout
|
|
|
|
// timer that will shut the connection down in case of no response.
|
|
|
|
SetPingTimer()
|
2020-01-21 14:26:08 +00:00
|
|
|
// SendVersion checks handshake status and sends a version message to
|
|
|
|
// the peer.
|
|
|
|
SendVersion() error
|
2019-09-13 12:43:22 +00:00
|
|
|
SendVersionAck(*Message) error
|
2020-01-15 14:03:42 +00:00
|
|
|
// StartProtocol is a goroutine to be run after the handshake. It
|
|
|
|
// implements basic peer-related protocol handling.
|
|
|
|
StartProtocol()
|
2019-09-13 12:43:22 +00:00
|
|
|
HandleVersion(*payload.Version) error
|
|
|
|
HandleVersionAck() error
|
2020-01-20 16:02:19 +00:00
|
|
|
|
2020-08-14 13:22:15 +00:00
|
|
|
// HandlePing checks ping contents against Peer's state and updates it.
|
|
|
|
HandlePing(ping *payload.Ping) error
|
|
|
|
|
2020-01-20 16:02:19 +00:00
|
|
|
// HandlePong checks pong contents against Peer's state and updates it.
|
|
|
|
HandlePong(pong *payload.Ping) error
|
2020-11-25 10:34:38 +00:00
|
|
|
|
|
|
|
// AddGetAddrSent is to inform local peer context that a getaddr command
|
|
|
|
// is sent. The decision to send getaddr is server-wide, but it needs to be
|
|
|
|
// accounted for in peer's context, thus this method.
|
|
|
|
AddGetAddrSent()
|
|
|
|
|
|
|
|
// CanProcessAddr checks whether an addr command is expected to come from
|
|
|
|
// this peer and can be processed.
|
|
|
|
CanProcessAddr() bool
|
2018-01-31 08:27:08 +00:00
|
|
|
}
|