Fix mempool and chain locking
This allows us easily make 1000 Tx/s in 4-nodes privnet, fixes potential
double spends and improves mempool testing coverage.
Fixes GolangCI:
Error return value of
(*github.com/CityOfZion/neo-go/pkg/core/mempool.Pool).Add is not checked
(from errcheck)
and allows us to almost completely forget about mempool here.
Our mempool only contains valid verified transactions all the time, it never
has any unverified ones. Unverified pool made some sense for quick unverifying
after the new block acceptance (and gradual background reverification), but
reverification needs some non-trivial locking between blockchain and mempool
and internal mempool state locking (reverifying tx and moving it between
unverified and verified pools must be atomic). But our current reverification
is fast enough (and has all the appropriate locks), so bothering with
unverified pool makes little sense.
We not only need to remove transactions stored in the block, but also
invalidate some potential double spends caused by these transactions. Usually
new block contains a substantial number of transactions from the pool, so it's
easier to make one pass over it only keeping valid items rather than remove
them one by one and make an additional pass to recheck inputs/witnesses.
It doesn't harm as we have transactions naturally ordered by fee anyway and it
makes managing them a little easier. This also makes slices store item itself
instead of pointers to it which reduces the pressure on the memory subsystem.
They shouldn't depend on the chain state and for the same transaction they
should always produce the same result. Thus, it makes no sense recalculating
them over and over again.
We can only add one block of the given height and we have two competing
goroutines to do that --- consensus and block queue. Whomever adds the block
first shouldn't trigger an error in another one.
Fix block relaying for blocks added via the block queue also, previously one
consensus-generated blocks were broadcasted.
Eliminate races between tx checks and adding them to the mempool, ensure the
chain doesn't change while we're working with the new tx. Ensure only one
block addition attempt could be in progress.
The chain may already be more current than our dBFT state (like when the node
has commited something at view 0, but all the other nodes changed view and
accepted something at view 1), so in this case we should reinit dBFT on new
height.
Because the constants are loaded directly via `emitLoadConst`, there is no need to store
them in an array of locals. It can have a big overhead, because it
is done at the beginning of every function.
It can lead to some goroutine explosion, but supposedly it's better than
stalling other processing and eventually all of these goroutines should finish
their sends. Note that this doesn't change the behavior for RPC-relayed
transactions that are still waiting for the broadcast to finish ensuring
proper transaction distribution before returning the result to the client.
If we have already got Version message, we don't need the rest of handshake to
complete before being able to properly answer the PeerAddr() requests. Fixes
some duplicate connections between machines.
This one is designed to give more priority to direct nodes communication, that
is that their messaging would have more priority than generic broadcasts. It
should improve consensus process under TX pressure and allow to handle
pings in time (preventing disconnects).
They have the opposite order, height first and nonce second. It was done wrong
in 4e6ed902 and never fixed since. Fixes sending wrong peer state leading to
useless getheaders messages (and disconnects when the other side is lagging
behind).
We can have more than one connection attempt in progress and not yet completed
the handshake, so if there is a Version already received we should look it.
Returning error string as a result (not an error) is utterly wrong, but C#
implementation just returns a zero balance for unknown addresses, so we should
follow that.
While decoding payload, local implementations of Recovery*
messages were used, but when creating RecoveryMessage inside dBFT
library default NewRecoveryMessage was invoked. This lead to parsing
errors.
Append should leave it's result on top of the stack.
Thus we need to transform top of the stack:
(top) a . b --> (top) a . b . b
It can be done with just OVER + SWAP.
Our node was too pingy because of wrong timer setups (that divided timeout
Duration by time.Second), it also was wrong in its time calculations (using
UTC time to calculate intervals). At the same time missing block is a
server-wide problem, so it's better solved with server-wide protocol loop.
A while ago VM serialization format for Integer items was changed
but compiler continued to emit Integers in old format.
This commit changes compiler behaviour to be compatible with VM.
Recursive execute() calls can affect gas calculation.
This commit makes execute() be called only for real opcodes
and moves duplicate logic for CALL/JMP into a separate function.
1) Make timeout a timeout, don't do magic ping counts.
2) Drop additional timer from the main peer's protocol loop, create it
dynamically and make it disconnect the peer.
3) Don't expose the ping counter to the outside, handle more logic inside the
Peer.
Relates to #430.
We don't and we won't have synchronized clocks in the network so the only
timestamp that we can compare our local time with is the one made
ourselves. What this ping mechanism is used for is to recover from missing the
block broadcast, thus it's appropriate for it to trigger after X seconds of
the local time since the last block received.
Relates to #430.
In reality it will never be true exactly in the case where we want this ping
mechanism to work --- when the node failed to get a block from the net. It
won't get the header either and thus its block height will be equal to header
height. The only moment when this condition is met is when the node does
initial synchronization and this synchronization works just fine without any
pings.
Relates to #430.
Two queues for high-priority and ordinary messages. Fixes#590. These queues
are deliberately made small to avoid buffer bloat problem, there is gonna be
another queueing layer above them to compensate for that. The queues are
designed to be synchronous in enqueueing, async capabilities are to be added
layer above later.
Big.Int Bytes()/SetBytes() methods are not symmetric.
Moreover we need to mimic C# node behavior:
- if a positive number has MSB set, 0x00 byte should be appended
to distinguish positive number from negatives
- negative numbers should serialize as two's-complement
add pingInterval same as used in ref C# implementation with the same logic
add pingTimeout which is used to check whether pong received. If not -- drop the peer.
add pingLimit which is hardcoded to 4 in TCPPeer. It's limit for unsuccessful ping/pong calls (where pong wasn't received in pingTimeout interval)
It wasn't actually requesting transactions but rather sending an inventory
message telling everyone that we have them which is completely wrong and
easily leads to ChangeView that could be avoided.
When system and network pressure is high it can be beneficial
to use transactions which and were already proposed.
The assumption is that they will be in other node's memory pool
with more probability.
If blockchain is not closed, logging in defer can occur
after test has finished, which will lead to a panic with
"Log in goroutine after Test* has completed".
There is no point in encoding the output of this function in a WIF format,
most of the users actually want the real key and those who need a WIF can
easily get if from the key (and it's simpler than getting the key from the
WIF).
It also fixes a severe bug in NEP2Decrypt, base58 decoding errors were not
processed correctly.
Error in Seek means something is terribly wrong (e.g. db was not opened) and
error drop is not the right thing to do, because caller
will continue working with the wrong view.
buildMerkleTree() is internal to the hash package and if anyone calls it with
`len(leaves) == 0` he deserves a panic. As it's the only error case in it, we
can remove error value return from this function and simplify NewMerkleTree().
Turns out, our dApps use it a lot and we were going to the DB to get it which
is a useless waste of time. Technically we could also remove blockHeight here,
but not doing it at the moment as it's more involved.
It eliminates this time waste from the pprof graph, but doesn't change 1.4M ->
1.5M 100K mainnet block import test case in any noticeable way.
Preseed the scriptHash value when we already know it. Eliminates this time
waste from the pprof graph, but doesn't really change anything in the 1.4M ->
1.5M 100K mainnet blocks import test.
These don't belong to VM as they compile some Go code and run it in a VM. One
may call them integration tests, but I prefer to attribute them to
compiler. Moving these tests into pkg/compiler also allows to properly count
the compiler coverage they add:
-ok github.com/CityOfZion/neo-go/pkg/compiler (cached) coverage: 69.7% of statements
+ok github.com/CityOfZion/neo-go/pkg/compiler (cached) coverage: 84.2% of statements
This change also fixes `contant` typo and removes fake packages exposed to the
public by moving foo/bar/foobar into the testdata directory.
This solves two problems:
* adds support for shortened SYSCALL form that uses IDs (similar to #434, but
for NEO 2.0, supporting both forms), which is important for compatibility
with C# node and mainnet chain that uses it from some height
* reworks interop plugging to use callbacks rather than appending to the map,
these map mangling functions are clearly visible in the VM profiling
statistics and we want spawning a VM to be fast, so it makes sense
optimizing it. This change moves most of the work to the init() phase
making VM setup cheaper.
Caveats:
* InteropNameToID accepts `[]byte` because that's the thing we have in
SYSCALL processing and that's the most often usecase for it, it leads to
some conversions in other places but that's acceptable because those are
either tests or init()
* three getInterop functions are: `getDefaultVMInterop`, `getSystemInterop`
and `getNeoInterop`
Our 100K (1.4M->1.5M) block import time improves by ~4% with this change.
Fix duping and add tests.
C# node actually implements DUP in the same way we did, but it does create a
new element when accessing some particular value (like BigInt() or Bytes()) so
in the end this DUP implementation doesn't lead to any visible side-effects. In
our case I think it's more appropriate to fix the DUP (and its variants) itself
avoiding useless allocations in the VM.
Add `Roll` method to Stack that doesn't pop and push values and use it for
ROLL and ROT.
1.4M->1.5M 100K block import test before:
real 3m44,292s
user 5m43,494s
sys 0m34,741s
After:
real 3m40,449s
user 5m42,701s
sys 0m35,500s
Add `Swap` method to the Stack and use it for both SWAP and XSWAP. Avoid
element popping and pushing (and associated accounting costs).
1.4M->1.5M 100K block import test before:
real 3m51,885s
user 5m54,744s
sys 0m38,444s
After:
real 3m44,292s
user 5m43,494s
sys 0m34,741s
First of all, it was wrong, it was not checking for inputs really, it compared
tx hashes for some reason, second, when it did compare inputs it compared only
the PrevIndex part of them which is also wrong.
Also, there is absolutely no reason to go through GetVerifiedTransactions()
here, we don't need this copy of pointers and it can also be outdated by the
time we're to finish our check.
Before:
BenchmarkTXPerformanceTest-4
5000 485506 ns/op 65886 B/op 409 allocs/op
ok github.com/CityOfZion/neo-go/integration 3.212s
After:
enchmarkTXPerformanceTest-4
5000 371104 ns/op 44367 B/op 408 allocs/op
ok github.com/CityOfZion/neo-go/integration 2.712s
This simple change improves our BenchmarkTXPerformanceTest by 14%, just
because we don't waste time on reallocations during append().
Before:
10000 439754 ns/op 218859 B/op 428 allocs/op
ok github.com/CityOfZion/neo-go/integration 5.423s
After:
10000 369833 ns/op 87209 B/op 412 allocs/op
ok github.com/CityOfZion/neo-go/integration 4.612s
Creating a new BinReader for every instruction is a bit too much and it adds
about 1% overhead on block import (and actually is quite visible in the VM
profiling statistics). So use a bit more ugly but efficient method.
It's useless work being done before it's actually needed. These (updated with
new values) are going to be written with some kind of Put anyway, so writing
them here is just a waste of time.
We're spending a lot of time here, 100K blocks import starting at 1.4M, before
this patch:
real 4m17,748s
user 6m23,316s
sys 0m37,866s
After:
real 3m54,968s
user 5m56,547s
sys 0m39,398s
9% is quite a substantial improvement to justify this change.
Importing 100K blocks starting at 1.4M, before this patch:
real 6m0,356s
user 8m52,293s
sys 0m47,372s
After this patch:
real 4m17,748s
user 6m23,316s
sys 0m37,866s
Almost 30% better.
Do not fill verification script randomly as there is a probability
for it to be executed sucessfully.
time="2019-12-12T17:24:22+03:00" level=info msg="blockchain persist completed" blockHeight=0 headerHeight=0 persistedBlocks=0 persistedKeys=15 took="54.474µs"
time="2019-12-12T17:24:23+03:00" level=info msg="blockchain persist completed" blockHeight=0 headerHeight=0 persistedBlocks=0 persistedKeys=15 took="49.312µs"
2019-12-12T17:24:24.026+0300 DEBUG can't verify payload from #%d1 {"module": "dbft"}
--- FAIL: TestPayload_Sign (0.00s)
payload_test.go:302:
Error Trace: payload_test.go:302
Error: Should be false
Test: TestPayload_Sign
FAIL
coverage: 75.8% of statements
FAIL github.com/CityOfZion/neo-go/pkg/consensus 2.145s
It's a getter function and even though it's quite fancy with its transactions
processing (for consensus operation) it shouldn't ever change the state of the
Blockchain. If we're to change anything here these changes may conflict with
the actual block processing later or may lead to broken state (if transactions
won't be approved for some reason).
go vet is not happy about them:
pkg/io/binaryReader.go:92:21: method ReadByte() byte should have signature ReadByte() (byte, error)
pkg/io/binaryWriter.go:75:21: method WriteByte(u8 byte) should have signature WriteByte(byte) error
This seriously improves the serialization/deserialization performance for
several reasons:
* no time spent in `binary` reflection
* no memory allocations being made on every read/write
* uses fast ReadBytes everywhere it's appropriate
It also makes Fixed8 Serializable just for convenience.
add dao which takes care about all CRUD operations on storage
remove blockchain state since everything is stored on change
remove storage operations from structs(entities)
move structs to entities package
This change (closely related to the neo-project/neo#1321 proposal) speeds up
1.4M mainnet blocks import by 30%. Basically, we're eliminating key decoding
for block's multisignature that has the same keys most of the time.
Things I don't like about this patch:
* yet another parameter for verifyHashAgainstScript()
* vm keys are not copied in/out
But it's rather simple and solves the problem for this particular case, so I
think it's worth it.
It can't be really solved in many cases (it's used in P2P protocol and we have
to follow the usual conventions there) and in most of the cases we don't care
about the difference between nil slice and zero-length slice.
It makes very little sense having pointers here, these structures MUST have
some kind of key and this key is not gonna be wandering somewhere on its
own. Fixes a part of #519.
It reduces heap pressure a little for these elements as we don't have to
allocate/free them individually. And they're directly tied to transactions or
block, not being shared or anything like that, so it makes little sense for
them to be pointer-based. It only makes building transactions a little easier,
but that's obviously a minor usecase.
reflect.MethodByName is a rather expensive function especially when
called on hot path. This became obvious during profiling of db restore.
This commit replaces reflection with a cast to an interface.
Before this patch on block import we could easily be spending more than 6
seconds out of 30 in Uint256 encoding for UnspentBalance, now it's completely
off the radar.
Which speeds it up at least twofold for a typical 32-bytes write (and that's
for a very naïve test that allocates new BufBinWriter on every iteration):
pkg: github.com/CityOfZion/neo-go/pkg/io
BenchmarkWriteBytes-8 10000000 124 ns/op
BenchmarkWriteBytesOld-8 5000000 251 ns/op
When 74590551 introduced this code we had no proper caching layer, so there
were these strange fallbacks in the code. fc0031e5 should'd removed them, but
failed to do so, so do it now and fix processing of transactions that touch
storage for the same key (address) in the same block.
To use opcode definitions you have to import whole vm package that you might
not care about at all. So this moves opcodes to their own package under vm, fixes
and deduplicate related code and moves compiler package up one level.
Drop wif.GetVerificationScript(), drop
smartcontract.CreateSignatureRedeemScript(), add GetVerificationScript()
directly to the PublicKey and use it everywhere.
This allows easier reuse of opcodes and in some cases allows to eliminate
dependencies on the whole vm package, like in compiler that only needs opcodes
and doesn't care about VM for any other purpose.
And yes, they're opcodes because an instruction is a whole thing with
operands, that's what context.Next() returns.
Only request headers from the other peer if his height is bigger than
ours. Otherwise we routinely ask 0-height newcomers for some random headers
that they know nothing about.
This one is essential for the consensus nodes as otherwise they won't give out
the blocks they generate making their generation almost useless. It also makes
our networking part more complete.
We have a race between reader and writer goroutines for the same connection
that leads to handshake failures when reader is faster to read the incoming
version (and try to reply to it) than writer is to write our own Version:
WARN[0000] peer disconnected addr="172.200.0.4:20334" peerCount=5 reason="invalid handshake: tried to send VersionAck, but didn't send Version yet
Fix it by moving Version sending before the reader loop starts.
Commit c80ee952a1 removed temporary store used
to contain changes of the block being processed. It's wrong in that the block
changes should be applied to the database in a single transaction so that
there wouldn't be any intermediate state observed from the outside (which is
possible now). Also, this made changes commiting persist them to the
underlying store effectively making our persist loop a no-op (and not
producing `persist completed` log lines that we love so much).
Param getters were redone to return errors because otherwise bad FuncParam
values could lead to panic. FuncParam itself might be not the most elegant
solution, but it works good enough for now.
This PR does 3 things:
adds array parameter unmarshalling
extend Param with convenient methods
refactor tests into using tables to make it easier add new tests
(part of #347 solution)
add processing of validators while block persist;
add validator structure with decoding/encoding;
add validator get from store;
add EnrollmentTX and StateTX processing;
add pubkey decode bytes, unique and contains functions;
Fixes failure to process transaction from the block when it was relayed
initially:
WARN[0788] blockQueue: failed adding block into the blockchain blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
WARN[0790] blockQueue: failed adding block into the blockchain blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
WARN[0790] blockQueue: failed adding block into the blockchain blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
Right now message can be written in several Write's so
concurrent calls of writeMsg() can in theory interleave.
This commit fixes it.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
When encountering already seen stack item we should fail
only if it is a collection. Duplicate Integers or ByteArrays are ok
because they can't lead to recursion.
If we're to receive some 500 headers (less than `headerBatchCount`) and quit
before receiving more of them we end up with clean `bc.headerList` that will
be inited going backwards to the `targetHash`, but code path doesn't add add
the `targetHash` itself which it should do in this particular case, otherwise
we end with no genesis block hash in the list.
Otherwise the node might crash in `startProtocol` because of missing Version
field in the peer. And it also keeps the sequence correct, Version MUST be
sent first and ACKs can only follow it.
Missing it the following line could fail on subsequent restarts:
currHeaderHeight, currHeaderHash, err := storage.CurrentHeaderHeight(bc.store)
if the node was stopped before any headers had been received.
VM: Use JSON-based tests from neoVM
After the implementation of stack limits nothing is needed for us to pass reference JSON tests :)
The only thing that differs --- we do not compare stack in case of FAULT (which matches NEO 3 behavior).
Also two commits were reverted to match 2.x VM behavior.
Our node didn't respect the MaxPeers setting, fix it with a drop of random
connection when this limit is reached (to give a chance for newcomers to
communicate), but also introduce AttemptConnPeers setting to tune the number
of attempted connections.
This also raises the default MaxPeers for testnet/mainnet to 100, because
neo-go nodes love making friends.
This allows to start handshaking from both client and server (mainnet/testnet
nodes were seen to not care about string ordering for it), but still maintains
some sane checks in the process. It also makes functions thread-safe because
we have two goroutines servicing read and write side of the Peer connection,
so they can clash on access to the struct fields.
Add a test for it also.
There is a difference in interpretation of what a block count is. neo-go nodes
currently respond to this request with the latest block number which is the
same number that neoscan.io shows. However, C# nodes deliberately do add one
to this number when answering to the getblockcount request to account for the
genesis block number 0.
This patch makes us consistent with C# nodes wrt to getblockcount behaviour.
This one enables our RPC to be called from the browser if there is a
need. It's insecure and not standards-compliant, thus this behaviour is
configurable is not enabled by default. It makes our node with this workaround
enabled compatible with neo-mon monitoring.
Originally debugged by @anatoly-bogatyrev in #464.
Extend Blockchainer with one more method to spawn a VM for test runs and use
it to run scripts. Gas consumption is not counted or limited in any way at the
moment (see #424).
Make inspect work with avms by default and with go files if told so. In the
end this makes our CLI interface more consistent and usable. Drop useless
CompileAndInspect() compiler method along the way.
Keeping run() as the owner of all maps would mean adding at least three more
channels to keep address getters with thread-safety. But then there also is a
race between requestToWork() and run() which is way harder to solve with
channels because there are lots of possibilities for deadlocks. So rework all
of this with good old mutexes.
While at it, fix `requestCh` handling in the inner select of run, it will waste
one loop to handle it, so we should add one to the `requested`.
Fixes#445.
Wrong bits were used to represent flags which is important for contracts
created via interop. Fixes contracts failing to store things:
WARN[16278] contract invocation failed block=3773025 err="error encountered at instruction 3435 (SYSCALL): failed to invoke syscall: contract c9d870d7857e956d82290d5df19de3133c107815 can't have storage" tx=fa695eea240b7b4dbb6f42ea6335447a764d8b629c40b7812ea3bca16b1f098d
WARN[16278] contract invocation failed block=3773025 err="error encountered at instruction 1279 (SYSCALL): failed to invoke syscall: contract 97210e7c98582151ceb37f9748c9a1d27d9ae6fd can't have storage" tx=0144d84038149fa0cf1f7912f7d5854fa5f3670f5b4217789c1441f9fd52d27b
NewInvocationTX() returned a version number one transaction that actually
failed to pass that version down to the invocation data which lead to
serialization/deserialization inconsistency.
VM should be responsible for code execution and in case anyone interested in additional logging or errors they could handle them like we do it iin cli.
When performing NEWARRAY on a Struct or NEWSTRUCT on a Array,
underlying slice needs to be copied, because when it's capacity
doesn't matches it's length, underlying storage will be used
for appends even if it is already pointed at by another slice.
We're about stored values here, so print those, which avoids blocking in
bc.HeaderHeight() and removes duplication between blockHeight and
persistedHeight. Fixes saving the blockchain on exit (deferred function in
Run() blocked in persist()).
Test modification was required because storeBlocks() doesn't actually save
headers and thus TestGetTransaction started to fail on persist().
If you're to sync less than 2000 headers no batched header key-value is
gonna be written into the DB and init() would panic because
bc.headerList.Len() would return 0. Use genesis block as a target in this
case.
Goreport:
neo-go/pkg/core/contract_state_test.go
Line 21: warning: "Contracto" is a misspelling of "Contraction" (misspell)
Line 64: warning: "Contracto" is a misspelling of "Contraction" (misspell)
neo-go/pkg/core/interop_neo.go
Line 420: warning: "succeedes" is a misspelling of "succeeds" (misspell)
neo-go/pkg/network/discovery.go
Line 118: warning: "succeded" is a misspelling of "succeeded" (misspell)
Line 128: warning: "successfuly" is a misspelling of "successfully" (misspell)
golint:
pkg/io/binaryrw_test.go:25:11: should omit type []byte from declaration of var bin; it will be inferred from the right-hand side
pkg/io/binaryrw_test.go:42:11: should omit type []byte from declaration of var bin; it will be inferred from the right-hand side
pkg/io/binaryrw_test.go:118:7: should omit type string from declaration of var str; it will be inferred from the right-hand side
golint suggests:
pkg/network/payload/address.go:48:12: should omit type net.IP from declaration of var netip; it will be inferred from the right-hand side
It's a temporary stub until proper encoding/decoding is implemented. It's
useful for testnet/mainnet connections because without it consensus message
receival leads to peer disconnection.
It's bogus and no other node implementation has anything like that. It fires
up for no good reason in the case when some other node connects to us and it
obviously doesn't use its listening port for it.
commit methods duplicated putSmthIntoStore functions, but have MemCachedStore
now that can easily substitute for a Batch, especially given that interop
needs something like that for its storage purposes anyway.
This adds the following verifications:
* merkleroot check
* index check
* timestamp check
* witnesses verification
VerifyWitnesses is also renamed to verifyTxWitnesses here to not confuse it
with verifyBlockWitnesse and to hide it from external access (no users at the
moment).
Linter isn't happy with our recent changes:
pkg/core/contract_state.go:109:1: receiver name cs should be consistent with previous receiver name a for ContractState
pkg/core/contract_state.go:114:1: receiver name cs should be consistent with previous receiver name a for ContractState
pkg/core/contract_state.go:119:1: receiver name cs should be consistent with previous receiver name a for ContractState
But actually `a` here most probably is a copy-paste from AssetState methods,
so fit the old code to match the new one.
Enable transaction verification for privnets and tests, testnet can't
successfuly verify block number 316711 with it enabled and mainnet stops at
105829.
We want to get a full block, so it has to have transactions
inside. Unfortunately our tests were used to this wrong behavior and utilized
completely bogus transactions without data that couldn't be persisted, so fix
that also.
PublishTX only had one of these flags, but newer contracts (created via the
interop function) can have more and these flags are aggregated into one field
that uses PropertyState enumeration (it's used to publish contract, so
supposedly it's also a nice choice for contract state storage).
It's used a lot and it looks a lot like MemoryStore, it just needs not to
return errors from Put and Delete, so make it use MemoryStore internally with
adjusted interface.
Make it look more like a real transaction, put/delete things with a single
lock. Make a copy of value in Put also, just for safety purposes, no one knows
how this value slice can be used after the Put.
Using pointers is just plain wrong here, because the batch can be updated with
newer values for the same keys.
Fixes Seek() to use HasPrefix also because this is the intended behavior.
Script can return non-bool results that can still be converted to bool
according to the usual VM rules. Unfortunately Bool() panics if this
conversion fails which is OK for things done in vm.execute(), but certainly
not for VerifyWitnesses(), thus there is a need for TryBool() that will just
return an error in this case.
It gives access to the internal value's Value() which is essential for interop
functions that need to get something from InteropItems. And it also simplifies
some already existing code along the way.
If the block references two ouputs in some other transaction the code failed
to verify it because of key collision. C# code implements it properly by using
full CoinReference type as a key, so let's do it in a similar fashion.
Claim transactions have different logic in C# node, so we need to
implement it too. It's not the most elegant way to fix it, but let's make it
work first and then refactor if and where needed. Fixes verification of Claim
transactions.
What started as an attempt to fix#366 ended up being quite substantial refactoring of the Blockchain->Store and Server->Blockchain interactions. As usually, some additional problems were noted and fixed along the way. It also accidentally fixes#410.
In the very specific case when the list of headers received is exactly one
block ahead of the chain of full blocks requestBlocks() failed to generate
request to get the next full block.
BoltDB doesn't have internal batching mechanism, thus we have a substitute for
it, but this substitute is absolutely identical to MemoryBatch, so it's better
to unify them and import ac5d2f94d3 fix into the
MemoryBatch.
Commit 578ac414d4 was wrong in that it saved
only a part of the block, so depending on how you use blockchain, you may
still see that the block was not really processed properly. To really fix it
this commit introduces intermediate storage layer in form of memStore, which
actually is a MemoryStore that supports full Store API (thus easily fitting
into the existing code) and one extension that allows it to flush its data to
some other Store.
It also changes AddBlock() semantics in that it only accepts now successive
blocks, but when it does it guarantees that they're properly added into the
Blockchain and can be referred to in any way. Pending block queing is now
moved into the server (see 8c0c055ac657813fe3ed10257bce199e9527d5ed).
So the only thing done with persist() now is just a move from memStore to
Store which probably should've always been the case (notice also that
previously headers and some other metadata was written into the Store
bypassing caching/batching mechanism thus leading to some inefficiency).
This one will replace blockCache in Blockchain itself as it can and should be
external from it. The idea is that we only feed successive blocks into the
Blockchain and it only stores valid proper Blockchain and nothing else.
This changes the Blockchain to also return unpersisted (theoretically, verified
in the AddBlock!) blocks and transactions, making Add/Get interfaces
symmetrical. It allows to turn Persist into internal method again and makes it
possible to enable transaction check in GetBlock(), thus fixing #366.
It must copy both the value and the key because they can be reused for other
purposes between Put() and PutBatch(). This actually happens with values in
headers processing, leading to wrong data being written into the DB.
Extend the batch test to check for that.
For example, at the moment our node can't handle `consensus` message, so when
it received it before the patch it just crashed because of uninitialized `p`.
earlier we had an issue with failing test in #353 and other one #305.
Reworked these test to have in-memory database. This led to multiple
changes: made some functions like Hash and Persist public(otherwise
it's not possible to control state of the blockchain); removed
unit_tests storage package which was used mainly for leveldb in unit
tests.
I see these tests not really good since they look like e2e tests and
as for me should be run in separate step against dockerized env or
in case we want to check rpc handler we might want to rework it in order
to have interface for proper unit tests.
As for me this patchset at least makes as safe with not removing totally
previous tests and at the same time CircleCI will be happy now.
It's mostly used for Serializable and in other cases where one needs to
estimate binary-encoded size of the stucture. This also simplifies future
removal of the Size() from Serializable.
The logic here is that we'll have all binary encoding/decoding done via our io
package, which simplifies error handling. This functionality doesn't belong to
util, so it's moved.
This also expands BufBinWriter with Reset() method to fit the needs of core
package.
add close function to storage interface
add common defer function call which will close db connection
remove context as soon as it's not needed anymore
updated unit tests
This one fixes#390 and some connected problems. After this patchset the node reconnects to some other nodes if anything goes wrong and it better senses when something goes wrong. It also fixes some block handling problems based on the testnet connection experience.
...and don't try to connect to the nodes we're already connected to.
Before this change we had a problem of discoverer throwing away good valid
addresses just because they are already known which lead to pool draining over
time (as address reuse was basically forbidden and getaddr may not get enough
new nodes).
Queuing one message is not reliable enough, the peer that gets it can fail to
actually make a request, so make this queue a bit deeper to have a higher
chance of success.
This makes writer side handle errors properly and fixes communication between
reader and writer goroutine to always correctly unregister the peer. This is
especially important for the case where error occurs before handshake
completes as in this case we don't even have goroutine in startProtocol()
running.
In the unlikely event of overlapping hash block written to the DB we might end
up with wrong hash list. That happened to me for some reason when synching
with the testnet leading to the following keys with respective values:
150000 -> 2000 hashes
152000 -> 2000 hashes
153999 -> 2000 hashes
Reading it hashes number 153999 and 154000 got the same values and the chain
couldn't sync correctly.
Same thing done in a2a8981979 for PUSHBYTES,
failing to read the amount of bytes specified should lead to FAULT. Also
makes readUint16() and readUint32() panic as this is the behavior we want in
these cases. Add some tests along the way.
Before:
NEO-GO-VM > loadgo h.go
READY: loaded 16 instructions
NEO-GO-VM > ip
instruction pointer at -1 (PUSH0)
After:
NEO-GO-VM > loadgo h.go
READY: loaded 16 instructions
NEO-GO-VM > ip
instruction pointer at -1 (NOP)
I think NOP is a little less scary.
Current NEO documentation lists them:
https://docs.neo.org/docs/en-us/tooldev/advanced/neo_vm.html
CALL_* instructions were left out because of conflict with golint (but they're
removed in NEO 3.0 anyway, so wasting time on them makes no sense).
Update autogenerated instruction_string.go accordingly.
The code that we have actually implements XTUCK and not TUCK. And it's a bit
broken, so fix it and add some tests. The most interesting one (that required
to touch stack code) is the one when we have 1 element on the stack and are
trying to tell XTUCK to push 2 elements deep.
ANSI X9.62 says that if x or y coordinate are greater than or equal to
curve.Params().P, the conversion should return an error (see ANSI X9.62:2005
Section A.5.8 Step b, which invokes Section A.5.5, which does the check and
rejects when x or y are too big.
See https://github.com/golang/go/issues/20482 for more details.
PublicKey() for PrivateKey now just can't fail and it makes no sense to return
an error from it. There is a lot of associated functionality for which this
also is true, so adjust it accordingly and simplify a lot of code.
Public key is just a point, so use the coordinates obtained previously to
initialize the PublicKey structure without jumping through the hoops of
encoding/decoding.
As NEO uses P256 we can use standard crypto/elliptic library for almost
everything, the only exception being decompression of the Y coordinate. For
some reason the standard library only supports uncompressed format in its
Marshal()/Unmarshal() functions. elliptic.P256() is known to have
constant-time implementation, so it fixes#245 (and the decompression using
big.Int operates on public key, so nobody really cares about that part being
constant-time).
New decompress function is inspired by
https://stackoverflow.com/questions/46283760, even though the previous one
really did the same thing just in a little less obvious way.
It makes no sense to provide an API for throw-away public keys, so obtain it
via a new real keypair generation where appropriate (and that's only needed
for testing).
Golint:
pkg/rpc/rpc.go:15:67: exported method GetBlock returns unexported type *rpc.response, which can be annoying to use
pkg/rpc/rpc.go:82:64: exported method GetRawTransaction returns unexported type *rpc.response, which can be annoying to use
pkg/rpc/rpc.go:97:52: exported method SendRawTransaction returns unexported type *rpc.response, which can be annoying to use
Refs. #213.
pkg/rpc/neoScanBalanceGetter.go:54:56: method parameter assetIdUint should be assetIDUint
pkg/rpc/neoScanBalanceGetter.go:62:3: var assetId should be assetID
pkg/rpc/server_test.go:27:5: var testRpcCases should be testRPCCases
pkg/rpc/txTypes.go:19:3: struct field assetId should be assetID
pkg/rpc/txTypes.go:39:35: interface method parameter assetId should be assetID
pkg/rpc/types.go:115:2: struct field TxId should be TxID
Refs. #213.
pkg/core/transaction/attribute.go:67:14: should omit type uint8 from declaration of var urllen; it will be inferred from the right-hand side
pkg/crypto/keys/publickey.go:184:8: should omit type []byte from declaration of var b; it will be inferred from the right-hand side
pkg/network/payload/version_test.go:15:12: should omit type bool from declaration of var relay; it will be inferred from the right-hand side
Refs. #213.
Golint:
pkg/core/blockchain.go:796:9: if block ends with a return statement, so drop
this else and outdent its block (move short variable declaration to its own
line if necessary)
Refs. #213.
Fixes things like:
* exported type/method/function X should have comment or be unexported
* comment on exported type/method/function X should be of the form "X ..."
(with optional leading article)
Refs. #213.
Fixes one more instruction being ran when VM FAULTs:
NEO-GO-VM > run
NEO-GO-VM > error encountered at instruction 6 (ROLL)
NEO-GO-VM > runtime error: invalid memory address or nil pointer dereference
FAULT
NEO-GO-VM > error encountered at instruction 7 (SETITEM)
NEO-GO-VM > interface conversion: interface {} is []vm.StackItem, not []uint8
Refs. #96.
And drop associated _pkg.dev remnants (refs. #307).
Original `dev` branch had two separate packages for public and private keys,
but those are so intertwined (`TestHelper` subpackage is a proof) that it's
better unite them and all associated code (like WIF and NEP-2) in one
package. This patch also:
* creates internal `keytestcases` package to share things with wallet (maybe
it'll be changed in some future)
* ports some tests from `dev`
* ports Verify() method for public key from `dev`
* expands TestPrivateKey() with public key check
Simplifies a lot of code and removes some duplication. Unfortunately I had to
move test_util random functions in same commit to avoid cycle
dependencies. One of these random functions was also used in core/transaction
testing, to simplify things I've just dropped it there and used a static
string (which is nice to have for a test anyway).
There is still sha256 left in wallet (but it needs to pass Hash structure into
the signing function).
Go's Hash is explicitly specified to never return an error on Write(), and our
own decoding functions only check for length which is gonna be right in every
case so it makes no sense returning errors from these functions.
With associated test and drop duplicating Uint160 implementation from
_pkg.dev. It doesn't seem to be used in pkg code at the moment, but still it
can be useful. Refs #307.
Unfortunately d58fbe0c88 didn't really fix the
problem because tinfo.Type (the expression resulting type) actually is a bool
and we need to check its parameters. Also, there is need to fix the NEQ
operation.
neo-storm has developed more wrappers for syscall APIs, so they can and should
be used as a drop-in replacement for pkg/vm/api. Moving it out of vm, as it's
not exactly related to the VM itself.
These were interpreted completely wrong, they actually have two next bytes
indicating an offset. This patch is a quick fix, actually more work is needed
here to properly display various instructions.
This is wrong, see issue #294, but it makes our VM tests work (as VM is
missing EQUAL implementation), so until #294 is properly resolved we're better
have this kind of wrong code generation.
* Fixed bug in else stmts
* Fixed if else bug
* Back to %v for formatting instructions
Imported from CityOfZion/neo-storm (ea8440e1454207753c8d209ce7c2cf724fd4ea16).
* Fixed bug where void functions cleaned up their mess multiple times
* Compiler improvements
* Fixed Neo APIs naming convention
* Added NOP after syscall for matching AVM files with neo-python.
Imported from CityOfZion/neo-storm (f93a28ef87d272a74f821d45ae0c6735a8c610be).
* Draft of iterator and enumerator
* Added iterator API to the syscall mapping
* Added draft of the enumerator.go file
* Added enumerator interop API.
* Updated the changelog
Imported from CityOfZion/neo-storm (156093318b8612e810965bb1ea26e1babfb46cdd).
* new mapping for interop api
* Fixed interop API mapping + added missing apis
* added engine apis
Imported from CityOfZion/neo-storm (ec5e6c8e2b587704a1e071e83b633d2d3a235300).
* Added new config attributes: 'SecondsPerBlock','LowPriorityThreshold'
* Added new files:
* Added new method: CompareTo
* Fixed empty Slice case
* Added new methods: LessThan, GreaterThan, Equal, CompareTo
* Added new method: InputIntersection
* Added MaxTransactionSize, GroupOutputByAssetID
* Added ned method: ScriptHash
* Added new method: IsDoubleSpend
* Refactor blockchainer, Added Feer interface, Verify and GetMemPool method
* 1) Added MemPool
2) Added new methods to satisfy the blockchainer interface: IsLowPriority, Verify, GetMemPool
* Added new methods: RelayTxn, RelayDirectly
* Fixed tests
* Implemented RPC server method sendrawtransaction
* Refactor getrawtransaction, sendrawtransaction in separate methods
* Moved 'secondsPerBlock' to config file
* Implemented Kim suggestions:
1) Fixed data race issues
2) refactor Verify method
3) Get rid of unused InputIntersection method due to refactoring Verify method
4) Fixed bug in https://github.com/CityOfZion/neo-go/pull/174#discussion_r264108135
5) minor simplications of the code
* Fixed minor issues related to
1) space
2) getter methods do not need pointer on the receiver
3) error message
4) refactoring CompareTo method in uint256.go
* Fixed small issues
* Use sync.RWMutex instead of sync.Mutex
* Refined (R)Lock/(R)Unlock
* return error instead of bool in Verify methods
* Added utility function GetVarSize
* 1) Added Size method: this implied that Fixed8 implements now the serializable interface. 2) Added few arithmetic operation (Add, Sub, div): this will be used to calculated networkfeeand feePerByte. Changed return value of the Value() method to int instead of int64. Modified fixed8_test accordingly.
* Implemented Size or MarshalJSON method.
- Structs accepting the Size method implement the serializable interface.
- Structs accepting the MarshalJSON method implements the customized json marshaller interface.
* Added fee calculation
* Implemented rcp server method GetRawTransaction
* Updated Tests
* Fixed:
1) NewFixed8 will accept as input int64
2) race condition affecting configDeafault, blockchainDefault
* Simplified Size calculation
* 1) Removed global variable blockchainDefault, configDefault
2) Extended Blockchainer interface to include the methods: References, FeePerByte, SystemFee, NetworkFee
3) Deleted fee_test.go, fee.go. Moved corresponding methods to blockchain_test.go and blockchain.go respectively
4) Amended tx_raw_output.go
* Simplified GetVarSize Method
* Replaced ValueAtAndType with ValueWithType
* Cosmetic changes + Added test case getrawtransaction_7
* Clean up Print statement
* Filled up keys
* Aligned verbose logic with the C#-neo implementation
* Implemented @Kim requests
Refactor server_test.go
* Small fixes
* Fixed verbose logic
Added more tests
Cosmetic changes
* Replaced assert.NoError with require.NoError
* Fixed tests by adding context.Background() as argument
* Fixed tests
* small fixes
* gofmt
* fix in raw tx build
* fixes after review
* balance getter interface
* moved address and signature calculation to public key
* errors handling
* PublicKey() returns PublicKey instead of bytes slice
* fixes after review
* fixes after review
* public key creation from asn1 serialized key
* small fixes
* gofmt
* fix in raw tx build
* fixes after review
* balance getter interface
* moved address and signature calculation to public key
* errors handling
* PublicKey() returns PublicKey instead of bytes slice
* fixes after review
* fixes after review
* Fix#130: Wrong answer for RPC server method getaccountstate
- fixed RPC Server response
- fixed RPC Server tests
- remove unused package from go.mod
* add index and type checker
* fix review comments (thx @aprasolova)
- balancer / wif / http.Client could be with data race
- add getter / setter with sync.Mutex
- now http.Client is pointer
- now you can provide your http.Client to rpcClient
* Implemented rcp method GetAccountState
* code clean up
* Removed empty line
* Used consistently github.com/pkg/errors package. Amended error message
* Get rid of fmt.Sprintf and use either errors.Errorf or errors.Wrapf
* cosmetic changes
* Fixed small incosistencies related to comments and variable name
* For testing purposes we have to move the method newBlockchain from the cli/server/server.go to pkg/core/blockchain.go. In addition we have to rename it to NewBlockchainLevelDB in order to be able to export it and avoid naming conflicts. In future we still need to think how to switch between different blockchain implementation easily but for the time being this is not possible.
* Added unit tests for the rpc server
* Added unit_testnet chain fixture
* fixed port number
* Added errors handling
* move unit_testnet chain from 'cli/chains' to 'pkg/rpc/chains'
* func to get private key from raw bytes
* added function to create raw transfer tx
* fixes
* more fixes
* prettify code and comments; neoscan interaction put in dedicated files
* added runtime serialize and deserialize functions
* removed getCurrentBlock from runtime functions
* Added block and header stdlib interop functions
* added transaction interop api
* added asset interop api
* bumped version
* Added missing storage.Find storage API function
* Fixed wrong example in the compiler README
* updated the compiler README to be more accurate on compiler features
* Initial draft of the ICO template ported from NEX.
* filled in token configuration
* added kyc storage prefix
* fixed byte array conversion + added tests
* fixed broken test + made 1 file for the token sale example.
* implemented the NEP5 handlers
* bumped version
* support VM to pass method and arguments to a script.
* added support for type assertions in smartcontracts.
* added native vm support for print.
* moved VM API packages to vm -> API
* reverted the native Print opcode in favor of runtime.Log
* added support for registering custom interop hooks in the VM.
* Updated README
* Updated compiler with @OPTIMIZE tags
* Moved more tests to VM package.
* optimized and refactored compiler and vm API
* updated README with new smartcontract apis
* bumped version
* Made Encode/Decode message public.
* Added Redis storage driver and made some optimizations for the initialising the blockchain
* removed log lines in tcp_peer
* Added missing comments on exported methods.
* bumped version
* renamed test folders and fixed bug where wrong jump labels would be exectuted for rewrite.
* Added support for Osize (len(string)) and factored out the array tests
* Added current instruction number to VM prompt if program is loaded.
* added support for unary expressions.
* updated README of and sorted the help commands
* updated readme of the compiler
* bumped version -> 0.39.0
* changed vm commands to match more of the standard
* fixed Uint16 jmp bug in VM
* moved test to vm + fixed numnotequal bug
* fixed broken tests
* moved compiler tests to vm tests
* added basic for support + inc and dec stmts
* bumped version
* Virtual machine for the NEO blockhain.
* fixed big.Int numeric operation pointer issue.
* added appcall
* Added README for vm package.
* removed main.go
* started VM cli (prompt) integration
* added support for printing the stack.
* moved cli to vm package
* fixed vet errors
* updated readme
* added more test for VM and fixed some edge cases.
* bumped version -> 0.37.0
* Few tweaks to improve output of `getblock`
* Adds few more fields and corrects witness
* Bumps version
* Only reverse when marshalling for moment
* Adds README for rpc package
* Few updates
* Typo
* Adds link in main readme
* Fix readme link
* implemented operation and param flags in the cli invoke cmd.
* reverted prev changes and added debug flag for compiling.
* change transactionType variable to Type, for package convention
* index support for arrays.
* implemented builtin (len) for the compiler.
* bumped version -> 0.35.0
* updated compiler README and changed invoke to testinvoke.
* Adds basic RPC supporting files
* Adds interrupt handling and error chan
* Add getblock RPC method
* Update request structure
* Update names of nodes
* Allow bad addresses to be registered in discovery externally
* Small tidy up
* Few tweaks
* Check if error is close error in tcp transport
* Fix tests
* Fix priv port
* Small tweak to param name
* Comment fix
* Remove version from server
* Moves submitblock to TODO block
* Remove old field
* Bumps version and fix hex issues
* added account_state + changed ECPoint to PublicKey
* account state persist
* in depth test for existing accounts.
* implemented GetTransaction.
* added enrollment TX
* added persist of accounts and unspent coins
* bumped version -> 0.32.0
* Created test_data folder with block json files for testing + create separate file for block base.
* Fixed bug in WriteVarUint + Trim logic + unit tests
* Refactored store and add more tests for it.
* restore headerList from chain file
* Fix tx decode bug + lots of housekeeping.
* Implemented Node restore state from chain file.
* Created standalone package for storage. Added couple more methods to Batch and Store interfaces.
* Block persisting + tests
* bumped version -> 0.31.0
* Adds struct for config settings
* Bumps version
* Move settings around
* Only push version tag up not whole branch as well
* Move config into own path and add test and main nets
* Update config for timeouts
* Convert to yaml
* block partial persist
* replaced refactored files with old one.
* removed gokit/log from deps
* Tweaks to not overburden remote nodes with getheaders/getblocks
* Changed Transporter interface to not take the server as argument due to a cause of race warning from the compiler
* started server test suite
* more test + return errors from message handlers
* removed --race from build
* Little improvements.
* Started RPC package to allow querying balances and sending raw transactions for sc's
* integrate invoke cmd in cli
* bumped version
* added sendrawtransaction to the rpc client.
* deleted transfer_output added asset type and transaction result to core
* removed writing 0x00 when buffer length is 0
* Refactored emit into VM package + moved tx to own package.
* implemented transaction along with claimTransaction.
* refactored naming of transaction + added decode address for uint160 types
* removed unnecessary folder and files.
* transaction/smartcontract logic
* bumped version 0.24.0
* Disconnect once go routines signal so
* Send msg if cmd is other than cmdUnknown
* Remove typo
* Fix lock file
* Updates README
* Comment typo
* Fix issue
* Revert old changes
* Handle error
* Initial draft of the neo-go wallet
* Cleanup + more test for util package
* integrated wallet into neo-cli partially
* base wallet implementation + smartcontract code.
* implemented global variables.
* refactored imports + lots and lots of other sweet stuff + fix#28.
* Implemented the VM interop runtime API (GetTrigger, CheckWitness, ...)
* Print proper error messages while using contract subcommands and also exit with status code 1.
* Make -in a flag, also remove dot from avm extension.
* Work on feedback by Anthony.
* Update README and VERSION
* implemented add, mul, div, sub assign for identifiers.
* Implemented struct field initialization.
* Implemented imports
* Implemented storage VM API (interop layer) + additional bug fixes when encountered.
* Bumped version 0.12.0
* fixed double point extension on compiled output file.
* Fixed bug where callExpr in returns where added to voidCall
* fixed binExpr compare equal
* Check the env for the gopath first
* removed travis.yml
* custom types + implemented general declarations.
* commented out the storage test to make the build pass
* refactor to use ast.Walk for recursive converting
* added lots of test cases
* added a new way to handle jump labels
* function calls with multiple arguments
* binary expression (LOR LAND)
* struct types + method receives
* cleaner opcode dumps, side by side diff for debugging test cases
* refactored tcp transport
* return errors on outgoing messages
* TCP transport should report its error after reading from connection
* handle error returned from peer transport
* bump version
* cleaned up error
