10 KiB
NeoGo support for neox (cross-chain Neo functionality)
NeoGo has full support for neox-2.x functionality integrated in the node, it doesn't require a separate build or code branch and it's completely controlled with two configuration options.
What is neox
Neox is an extension of original Neo 2 node originally implemented in neox-2.x branch of C# implementation (and then presented in the 2.11.0 official release). It includes the following main changes:
- local state root generation for contract storages based on MPT
- consensus updates for state root exchange between CNs and generation of verified (signed by CNs) state root
- P2P protocol updates for state root distribution
- RPC protocol updates for state status data and proofs generation
- two new key recovery syscalls for smart contracts
Most of these changes are pure extensions to Neo 2 protocol, but consensus changes are incompatible with regular Neo 2 nodes. The idea is that we have now some state reference for each block that can be used by other chains (along with proof paths for individual key-value pairs if needed) and at the same time we're able to check non-Neo signatures using new key recovery functionality that is available for two curves: Secp256r1 and Secp256k1.
How local state is being generated and what it covers
Any full node processing blocks can now generate state root information
locally using Merkle Patricia Trie (MPT). It's used for any key-value pairs
stored in the database with prefix of ST_Storage
which is used for contracts
data storage. Basically, anything contracts save using Neo.Storage.Put
syscall gets accounted for.
Each value gets a leaf node in MPT and the key for that value is encoded in branch and extension nodes according to prefix data. Any node in MPT can be hashed and the root node hash naturally depends on every other hash in the trie, so this single hash value represents current state of the trie and is called state root hash. Any change to the trie state (adding/deleting/changing key-value pairs) changes state root hash.
But even though this state root data can be computed at every full node it can't be considered authoritative until it's signed by network-trusted entities which are consensus nodes.
How and why consensus process was changed in neox
Consensus nodes now exchange state root signatures for height N-1 during consensus process for block N with PrepareRequest and PrepareResponse messages.
If all goes well CNs collect enough signatures for this state root data and
generate (and broadcast) a stateroot
message along with regular Commit
consensus messages. Effectively this creates another signed chain on the
network that is always one block behind from the main chain because the
process of block N
creation confirms the state resulting from processing of
block N - 1
.
How P2P protocol was changed
P2P protocol was extended with getroots
, roots
and stateroot
messages for state root data exchange. Simple stateroot
message is what
consensus nodes generate to broadcast signed state root data, it's accepted by
all nodes, they check it, verify its signature and save locally (to do that
they have to have confirmed state root for the previous block). It's somewhat
similar to block announcement, but as this message is rather small, inv
is
not being used.
But this message might get lost or some new node may join the network and want
to get verification for its state, so there has to be some possibility for
state root requests and replies and that's what getroots
/roots
pair is
for. In general it's expected that the node would synchronize state roots the
same way it synchronizes blocks, always trying to be up to date with the
network. From this synchronization comes the concept of "state height" which
represents the latest verified state root known to the node.
How RPC protocol was changed
RPC got extended with four new methods: getproof
, getstateheight
,
getstateroot
and verifyproof
.
getstateheight
and getstateroot
are easy, the first one allows to get
current node's block and state heights, while the second one returns state
root data for the specified (by index or by hash) block. State root data
basically mirrors the one exchanged via P2P protocol (version, previous state
root message hash and current state root hash), but also contains an
additional flag to specify if the node has a verification (signature) for this
state root. If the state is verified then the node also includes witness data
for this state root which use the same format transaction's witnesses use.
getproof
and verifyproof
methods are a bit more special as they allow you
to prove that some key-value pair exists in Neo state DB without having whole
state DB (like when you're operating on a different chain or when you're
working as a light node). This works via MPT path encoding from the root node
to the particular leaf (value) node you're interested in (that contains some
token balance for example). Using this path data it's easy to regenerate a
part of MPT corresponding to that key-value pair locally and recalculate
MPT hashes for that trie. If the top-level hash matches verified root hash
then you have a proof that the key-value pair is a part of the state DB shared
by all proper Neo nodes.
So getproof
method returns this path from the root node to the given
key. It can then be used to verify the proof locally or can be used to send
this proof to some trusted RPC node to verify it using verifyproof
method
that returns value for that key in case of success.
What are these new neox syscalls
Two syscalls were added along with other neox changes: "Neo.Cryptography.Secp256k1Recover" and "Neo.Cryptography.Secp256r1Recover", they're similar in their function and interface, but using different elliptic curves for their operation. The first one uses SEC-standardized Koblitz curve widely known for its usage in Bitcoin and the second one operates on regular SEC-standardized curve that is used by Neo.
Both of these syscalls allow to recover public key from the given signature (r, s) on the given message hash with a help of a flag denoting Y's least significant bit in decompression algorithm. The return value is a byte array representing recovered public key (64 bytes containing 32-byte X and Y) in case of success and zero-length byte array in case of failure.
This functionality allows you to check message signatures in smart contract, the key recovered can be compared with an expected one or be hashed and compared with an expected key hash (depending on what data is provided by the other blockchain).
How neox is supported in NeoGo
NeoGo has full support for functionality outlined above. Syscalls are
available via interop wrappers in crypto
packages and RPC client contains
methods to work with new RPC protocol extensions. Client-side support is
always available, but NeoGo node's behavior is controlled by two configuration
options: EnableStateRoot and StateRootEnableIndex, the first one is boolean
and the second one is integer. If not specified in the configuration the first
one has a default of false and the second has a default value of 0.
EnableStateRoot controls state root generation and processing functionality. NeoGo is able to operate both on stateroot-enabled and classic networks, so this is the main switch between these two modes.
With EnableStateRoot set to false the node works in classic mode:
- no local state root is being generated
- consensus process operates using classic message formats not including state root data
- stateroot-related P2P messages are ignored
- stateroot-related RPC calls are available, but always return an error
- recovery syscalls are unavailable to contracts
- StateRootEnableIndex setting is ignored
With EnableStateRoot set to true things change and the node operates with full
neox support, but a StateRootEnableIndex setting may additionally affect its
P2P-processing behavior. getroots
requests for blocks with height less than
StateRootEnableIndex are ignored, roots
messages are only processed for
blocks higher than StateRootEnableIndex and the node doesn't actively try to
synchronize its state height until its block height reaches
StateRootEnableIndex. This setting is made for network upgrades when there are
no confirmed state roots for old blocks and they'll never be properly
confirmed.
Things you can do
Running a classic network
Doesn't require changing anything, just upgrade the node and run it.
Running new stateroot-enabled network
Setting EnableStateRoot to true and not setting StateRootEnableIndex is a good choice for a new private network as it gives you all the functionality from block zero. Note that all consensus nodes must be using this settings combination for successful operation.
Adding stateroot functionality to existing network
If you already have some network and you need it to continue working, but want to upgrade it with neox functionality you need to:
- prepare a current dump of network's blocks
- upgrade all consensus nodes with NeoGo 0.76.0+
- stop all of them
- change their configuration, setting EnableStateRoot to true and StateRootEnableIndex to some block in the future (not far away from current network's height)
- remove CNs local databases
- import blocks from the previously generated dump on all CNs
- start all CNs
This can be optimized to reduce network's downtime by doing block dumps/restores with old CNs still running, but you have to regenerate local databases with stateroot enabled for correct operation.
Things you shouldn't do
Randomly changing EnableStateRoot setting
Switching EnableStateRoot on and off without full block resynchronization may lead to unexpected results on any full node (independent of whether it's a consensus node or not) because with EnableStateRoot set to true an MPT structure is initialized using local DB and if that DB doesn't have correct MPT state it will fail. If you're changing this setting in any way --- restore the DB from block dump.
Running mixed consensus nodes set
All consensus nodes should agree on the protocol being used, either all of them use state roots, or all of them don't. Mixing two types of nodes will lead to consensus failures.