NEO VM does not distinguish between empty and nil slices. Supporting
this is not easy and requires changing lots of other opcodes.
Pointers are not supported anyway and slices can be checked for
emptiness by inspecting their `len`.
Set default value also for complex (struct, map) types.
Note: because VM does not distinguish empty array and nil array,
comparison 'a == nil' can't be handled properly without substantial
effort. This will be fixed in NEO3.
Using view number from the recovery message is just plain wrong, it's gonna be
higher than our current view and these messages will be treated as coming from
the future, even though they have their original view number included.
Implement (*Param).GetBoolean() for converting parameter to bool value.
It is used for verbosity flag and is false iff it is either zero number
or empty sting.
We need to compact our in-memory MPT from time to time, otherwise it quickly
fills up all available memory. This raises two obvious quesions --- when to do
that and to what level do that.
As for 'when', I think it's quite easy to use our regular persistence interval
as an anchor (and it also frees up some memory), but we can't do that in the
persistence routine itself because of synchronization issues (adding some
synchronization primitives would add some cost that I'd also like to avoid),
so do it indirectly by comparing persisted and current height in `storeBlock`.
Choosing proper level is another problem, but if we're to roughly estimate one
full branch node to use 1K of memory (usually it's way less than that) then we
can easily store 1K of these nodes and that gives us a depth of 10 for our
trie.
Items were serialized several times if there were several successful
transactions in a block, prevent that by using State field as a bitfield (as
it almost was intended to) and adding one more bit. It also eliminates useless
duplicate MPT traversions.
Confirmed to not break storage changes up to 3.3M on testnet.
This was differing from C# notion of PrevHash. It's not a previous root, but
rather a hash of the previous serialized MPTRoot structure (that is to be
signed by CNs).
Implement secp256k1 and secp256r1 recover interops, closes#1003.
Note:
We have to implement Koblitz-related math to recover keys properly
with Neo.Cryptography.Secp256k1Recover interop as far as standard
go elliptic package supports short-form Weierstrass curve with a=-3
only (see https://github.com/golang/go/issues/26776 for details).
However, it's not the best choise to have a lot of such math in our
project, so it would be better to use ready-made solution for
Koblitz-related cryptography.
Because trie size is rather big, it can't be stored in memory.
Thus some form of caching should also be implemented. To avoid
marshaling/unmarshaling of items which are close to root and are used
very frequenly we can save them across the persists.
This commit implements pruning items at the specified depth,
replacing them by hash nodes.
There is nothing wrong with iterators being implemented in other parts
of code (e.g. Storage.Find). In this case type assertions can
prevent bugs at compile-time.
Reproduce behavior of the reference realization:
- if item was Put in cache after it was encountered during
Storage.Find, it must appear twice
- checking if item is in cache must be performed in real-time
during `Iterator.Next()`
bafdb916a0 change was wrong (probably brought
from neo-vm 3.0 at the state at which it existed back then), neo-vm 2.x
doesn't allow PICKITEM for arbitrary types.
The order in which storage.Find items are returns depends on what items
were processed in previous transactions of the same block.
The easiest way to implement this sort of caching is to cache operations
with storage, flushing the only in `Persist()`.
It's useless. Even though there is Neo.Transaction.GetUnspentCoins syscall
that can be used, its return type is an interop structure that's not accepted
by any other syscall, so you can't really do anything with it. And there is no
such interface for the .net Framework.
Previously we could generate dynamic appcall with a kludge of
AppCall([]byte{/* 20 zeroes */, realScriptHash, args...)
Now there is a separate function for this.
This syscall should only work for contracts created by current transaction and
that is what is supposed to be checked here. Do so by looking at the
differences between ic.dao and original lower DAO.
Our block.Block was JSONized in a bit different fashion than result.Block in
its Nonce and NextConsensus fields. It's not good for notifications because
third-party clients would probably expect to see the same format. Also, using
completely different Block representation in result is probably making our
client a bit weaker as this representation is harder to use with other neo-go
components.
So use the same approach we took for Transactions and wrap block.Base which is
to be serialized in proper way.
Getting batch, updating Prometheus metrics and pushing events doesn't require
any locking: batch is a local cache batch that no one outside cares about,
Prometheus metrics are not critical to be in perfect sync and events are
asynchronous anyway.
Note that the protocol differs a bit from #895 in its notifications format,
to avoid additional server-side processing we're omitting some metadata like:
* block size and confirmations
* transaction fees, confirmations, block hash and timestamp
* application execution doesn't have ScriptHash populated
Some block fields may also differ in encoding compared to `getblock` results
(like nonce field).
I think these differences are unnoticieable for most use cases, so we can
leave them as is, but it can be changed in the future.
We actually have to do that in order to answer getapplicationlog requests for
transactions that leave some interop items on the stack. It follows the same
logic our binary serializer/deserializes does leaving the type and stripping
the value (whatever that is).
It will be important for proper subscription testing and it doesn't hurt even
though technically we've got two http servers listening after this change (one
is a regular Server's http.Server and one is httptest's Server). Reusing
rpc.Server would be nice, but it requires some changes to Start sequence to
start Listener with net.Listen and then communicate back its resulting
Addr. It's not very convenient especially given that no other code needs it,
so doing these changes just for a bit cleaner testing seems like and
overkill.
Update config appropriately. Update Start comment along the way.
Get new blocks directly from the Blockchain. It may lead to some duplications
(as we'll also receive our own blocks), but at the same time it's more
correct, because technically we can also get blocks via other means besides
network server like RPC (submitblock call). And it simplifies network server
at the same time.
A part of integration with NEO Blockchain Toolkit (see #902). To be
able to deploy smart-contract compiled with neo-go compiler via NEO
Express, we have to generate additional .abi.json file. This file
contains the following information:
- hash of the compiled contract
- smart-contract metadata (title, description, version, author,
email, has-storage, has-dynamic-invoke, is-payable)
- smart-contract entry point
- functions
- events
However, this .abi.json file is slightly different from the one,
described in manifest.go, so we have to add auxilaury stractures for
json marshalling. The .abi.json format used by NEO-Express is described
[here](https://github.com/neo-project/neo-devpack-dotnet/blob/master/src/Neo.Compiler.MSIL/FuncExport.cs#L66).
Method `methodInfoFromScope(...)` always returned an empty parameters
set, so we were missing this information in both .abi.json and
.debug.json files. Fixed now.
Sequence points is a way to map a specific instruction offset
from a compiled contract to a text span in a source file.
This commit implements mapping only for `return` statements.
Further improvements are straight-forward.
There's a bug after #785: smartcontract.Parameter of type hash160 should
be marshalled in LE (as default marshaller for uint160 does) instead of
BE, so fixed.
Which makes iterating over map stable which is important for serialization and
and even fixes occasional test failures. We use the same ordering here as
NEO 3.0 uses, but it should also be fine for NEO 2.0 because it has no
defined order.
Fixes#809.
Basically, there are three alternative approaches to fixing it:
* allowing both []byte and string for ByteArrayType value
minimal invasion into existing code, but ugly as hell and will probably
backfire at some point
* storing string values in ByteArrayType
incurs quite a number of type conversions (and associated data copying),
though note that these values are not changed usually, so dynamic
properties of []byte are almost irrelevant here
* storing only []byte values in ByteArrayType
makes it impossible to use them as map keys which can be solved in several
ways:
- via an interface (Marshalable)
which is good, but makes testing and comparing values in general harder,
because of keys mismatch
- using serialized Parameter as a key (in a string)
which will need some additional marshaling/unmarshaling
- converting MapType from map to a slice of key-value pairs
not a bad idea as we don't use this map as a map really, the type
itself is all about input/output for real VM types and this approach is
also a bit closer to JSON representation of the Map
C# pushes value and key to the stack of non-serialized items, so key gets
serialized first followed by value. Fixes#806.
Notice though that neither IDictionary in C#, nor map in Go have elements
ordered, so we can easily get a difference in KV pairs order and it would be
impossible to fix.
Most of the time it's persisted into the MemoryStore or MemCachedStore, when
that's the case there is no real need to go through the Batch mechanism as it
incurs multiple copies of the data.
Importing 1.5M mainnet blocks with verification turned off, before:
real 12m39,484s
user 20m48,300s
sys 2m25,022s
After:
real 11m15,053s
user 18m2,755s
sys 2m4,162s
So it's around 10% improvement which looks good enough.
Previously, struct variables were initialize with VM's nil value
which is of primitive type. Thus SETITEM used for struct's field
updating wasn't working.
Previously this declarations were ignored which resulted
in runtime errors, because VM's nil is an element of primitive type
and can't be converted to an array.
Frequently one needs to check if struct serializes/deserializes
properly. This commit implements helpers for such cases including:
1. JSON
2. io.Serializable interface
