From e97c153a268609f162172221110e7b9ba8100b01 Mon Sep 17 00:00:00 2001 From: Roman Khimov Date: Thu, 25 Jun 2020 17:05:10 +0300 Subject: [PATCH] docs: add neox documentation --- README.md | 3 +- docs/neox.md | 210 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 docs/neox.md diff --git a/README.md b/README.md index 29e761596..924d097be 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,8 @@ A complete toolkit for the NEO blockchain, including: - [NEO virtual machine](docs/vm.md) This branch (**master-2.x**) is a stable version of the project compatible -with Neo 2, it only receives bug fixes and minor updates. For the Neo 3 +with Neo 2 (including [cross-chain neox support](docs/neox.md)), it only +receives bug fixes and minor updates. For Neo 3 development version please refer to the [**master** branch](https://github.com/nspcc-dev/neo-go/tree/master) and releases after 0.90.0. Releases before 0.80.0 (**0.7X.Y** track) are made from this diff --git a/docs/neox.md b/docs/neox.md new file mode 100644 index 000000000..d45434143 --- /dev/null +++ b/docs/neox.md @@ -0,0 +1,210 @@ +# NeoGo support for neox (cross-chain Neo functionality) + +NeoGo has full support for neox-2.x functionality integrated in the node, it +doesn't require a separate build or code branch and it's completely controlled +with two configuration options. + +## What is neox + +Neox is an extension of original Neo 2 node implemented in neox-2.x branch of +C# implementation. It includes the following main changes: + * local state root generation for contract storages based on MPT + * consensus updates for state root exchange between CNs and generation of + verified (signed by CNs) state root + * P2P protocol updates for state root distribution + * RPC protocol updates for state status data and proofs generation + * two new key recovery syscalls for smart contracts + +Most of these changes are pure extensions to Neo 2 protocol, but consensus +changes are incompatible with regular Neo 2 nodes. The idea is that we have +now some state reference for each block that can be used by other chains +(along with proof paths for individual key-value pairs if needed) and at the +same time we're able to check non-Neo signatures using new key recovery +functionality that is available for two curves: Secp256r1 and Secp256k1. + +### How local state is being generated and what it covers + +Any full node processing blocks can now generate state root information +locally using Merkle Patricia Trie (MPT). It's used for any key-value pairs +stored in the database with prefix of `ST_Storage` which is used for contracts +data storage. Basically, anything contracts save using `Neo.Storage.Put` +syscall gets accounted for. + +Each value gets a leaf node in MPT and the key for that value is encoded in +branch and extension nodes according to prefix data. Any node in MPT can be +hashed and the root node hash naturally depends on every other hash in the +trie, so this single hash value represents current state of the trie and is +called state root hash. Any change to the trie state +(adding/deleting/changing key-value pairs) changes state root hash. + +But even though this state root data can be computed at every full node it +can't be considered authoritative until it's signed by network-trusted +entities which are consensus nodes. + +### How and why consensus process was changed in neox + +Consensus nodes now exchange state root information with PrepareRequest +messages, so the Primary node tells everyone its current state root hash +(along with the block index that state root corresponds to) and the hash of +the previous state root message. This data might also be versioned in case of +future updates, so there is a special field reserved for that too, but at the +moment it's always 0. Backups either confirm this data (if it matches their +local state) by proceeding with PrepareResponse or request a ChangeView if +there is some mismatch detected. + +If all goes well CNs generate a signature for this state root data and +exchange it with their Commit messages (along with new block +signatures). Effectively this creates another signed chain on the network that +is always one block behind from the main chain because the process of block `N` +creation confirms the state resulting from processing of block `N - 1`. A +separate `stateroot` message is generated and sent along with the new block +broadcast. + +### How P2P protocol was changed + +P2P protocol was extended with `getroots`, `roots` and `stateroot` +messages for state root data exchange. Simple `stateroot` message is what +consensus nodes generate to broadcast signed state root data, it's accepted by +all nodes, they check it, verify its signature and save locally (to do that +they have to have confirmed state root for the previous block). It's somewhat +similar to block announcement, but as this message is rather small, `inv` is +not being used. + +But this message might get lost or some new node may join the network and want +to get verification for its state, so there has to be some possibility for +state root requests and replies and that's what `getroots`/`roots` pair is +for. In general it's expected that the node would synchronize state roots the +same way it synchronizes blocks, always trying to be up to date with the +network. From this synchronization comes the concept of "state height" which +represents the latest verified state root known to the node. + +### How RPC protocol was changed + +RPC got extended with four new methods: `getproof`, `getstateheight`, +`getstateroot` and `verifyproof`. + +`getstateheight` and `getstateroot` are easy, the first one allows to get +current node's block and state heights, while the second one returns state +root data for the specified (by index or by hash) block. State root data +basically mirrors the one exchanged via P2P protocol (version, previous state +root message hash and current state root hash), but also contains an +additional flag to specify if the node has a verification (signature) for this +state root. If the state is verified then the node also includes witness data +for this state root which use the same format transaction's witnesses use. + +`getproof` and `verifyproof` methods are a bit more special as they allow you +to prove that some key-value pair exists in Neo state DB without having whole +state DB (like when you're operating on a different chain or when you're +working as a light node). This works via MPT path encoding from the root node +to the particular leaf (value) node you're interested in (that contains some +token balance for example). Using this path data it's easy to regenerate a +part of MPT corresponding to that key-value pair locally and recalculate +MPT hashes for that trie. If the top-level hash matches verified root hash +then you have a proof that the key-value pair is a part of the state DB shared +by all proper Neo nodes. + +So `getproof` method returns this path from the root node to the given +key. It can then be used to verify the proof locally or can be used to send +this proof to some trusted RPC node to verify it using `verifyproof` method +that returns value for that key in case of success. + +### What are these new neox syscalls + +Two syscalls were added along with other neox changes: +"Neo.Cryptography.Secp256k1Recover" and "Neo.Cryptography.Secp256r1Recover", +they're similar in their function and interface, but using different elliptic +curves for their operation. The first one uses SEC-standardized Koblitz curve +widely known for its usage in Bitcoin and the second one operates on regular +SEC-standardized curve that is used by Neo. + +Both of these syscalls allow to recover public key from the given signature +(r, s) on the given message hash with a help of a flag denoting Y's least +significant bit in decompression algorithm. The return value is a byte +array representing recovered public key (64 bytes containing 32-byte X and Y) +in case of success and zero-length byte array in case of failure. + +This functionality allows you to check message signatures in smart contract, +the key recovered can be compared with an expected one or be hashed and +compared with an expected key hash (depending on what data is provided by the +other blockchain). + +## How neox is supported in NeoGo + +NeoGo has full support for functionality outlined above. Syscalls are +available via interop wrappers in `crypto` packages and RPC client contains +methods to work with new RPC protocol extensions. Client-side support is +always available, but NeoGo node's behavior is controlled by two configuration +options: EnableStateRoot and StateRootEnableIndex, the first one is boolean +and the second one is integer. If not specified in the configuration the first +one has a default of false and the second has a default value of 0. + +EnableStateRoot controls state root generation and processing +functionality. NeoGo is able to operate both on stateroot-enabled and classic +networks, so this is the main switch between these two modes. + +With EnableStateRoot set to false the node works in classic mode: + * no local state root is being generated + * consensus process operates using classic message formats not including + state root data + * stateroot-related P2P messages are ignored + * stateroot-related RPC calls are available, but always return an error + * recovery syscalls are unavailable to contracts + * StateRootEnableIndex setting is ignored + +With EnableStateRoot set to true things change and the node operates with full +neox support, but a StateRootEnableIndex setting may additionally affect its +P2P-processing behavior. `getroots` requests for blocks with height less than +StateRootEnableIndex are ignored, `roots` messages are only processed for +blocks higher than StateRootEnableIndex and the node doesn't actively try to +synchronize its state height until its block height reaches +StateRootEnableIndex. This setting is made for network upgrades when there are +no confirmed state roots for old blocks and they'll never be properly +confirmed. + +### Things you can do + +#### Running a classic network + +Doesn't require changing anything, just upgrade the node and run it. + +#### Running new stateroot-enabled network + +Setting EnableStateRoot to true and not setting StateRootEnableIndex is a good +choice for a new private network as it gives you all the functionality from +block zero. Note that all consensus nodes must be using this settings +combination for successful operation. + +#### Adding stateroot functionality to existing network + +If you already have some network and you need it to continue working, but want +to upgrade it with neox functionality you need to: + * prepare a current dump of network's blocks + * upgrade all consensus nodes with NeoGo 0.76.0+ + * stop all of them + * change their configuration, setting EnableStateRoot to true and + StateRootEnableIndex to some block in the future (not far away from current + network's height) + * remove CNs local databases + * import blocks from the previously generated dump on all CNs + * start all CNs + +This can be optimized to reduce network's downtime by doing block +dumps/restores with old CNs still running, but you have to regenerate local +databases with stateroot enabled for correct operation. + +### Things you shouldn't do + +#### Randomly changing EnableStateRoot setting + +Switching EnableStateRoot on and off without full block resynchronization may +lead to unexpected results on any full node (independent of whether it's a +consensus node or not) because with EnableStateRoot set to true an MPT +structure is initialized using local DB and if that DB doesn't have correct +MPT state it will fail. If you're changing this setting in any way --- restore +the DB from block dump. + +#### Running mixed consensus nodes set + +All consensus nodes should agree on the protocol being used, either all of +them use state roots, or all of them don't. Mixing two types of nodes will +lead to consensus failures.