There are 2 problems:
1. `getvalidators` RPC can return empty list.
2. `+1` in single node can be too resrictive.
Proper solution for (1) may require requesting
standby validators. Here we add constant
to fix occasional test failures.
If port is dynamically allocated, `(*Server).Addr` will contain
0 port. This commit executes listener before exiting from `Start()`
and sets Addr to the actual address.
There is a notification pushed into the channel when block is being added,
that notification is read by special goroutine that then forwards it to all
subscribers to that particular event. Consensus goroutine is one of that
subscribers, so for the system to properly function it has to read these
events, but at the same time it can generate new blocks inside, so in some
cases it can generate two blocks without ever reading from the subscription
channel and this will lead to a deadlock.
To avoid that we need to check subscription channel for events on every loop.
It can't ever happen. We're guaranteed to have a consistent chain of headers
(we're verifying them above, if we're not verifying --- it's not our fault)
that starts at HeaderHeight that was actual when we were asking for it
previously. HeaderHeight can only move forward, so if that happened that would
be filtered out by the condition below and the first one can't happen. Though
to be absolutely sure change the second check to only pass "+1" headers (which
is what we want).
It's used in two places now:
* Blockchain.AddBlock()
This one does transaction duplication check of its own, doing it in
Verify() is just a waste of time. Merkle tree root hash value check is
still relevant though
* Block.DecodeBinary()
We're decoding blocks for the following purposes:
- on restore from dump
The block will be added to the chain via AddBlock() and that will do a
full check of it (if configured to do so)
- on retrieving the block from the DB (DAO)
We trust the DB, if it's gone wild, this check won't really help
- on receiving the block via P2P
It's gonna be put into block queue and then end up in AddBlock() which
will check it
- on receiving the block via RPC (submitblock)
It is to be passed into AddBlock()
- on receiving the block via RPC in a client
That's the only problematic case probably, but RPC client has to trust
the server and it can check for the signature if it really
cares. Or a separate in-client check might be added.
As we can see nothing really requires this verification to be done the way it
is now, AddBlock can just have a Merkle check and DecodeBinary can do fine
without it at all.
It's a no-op and there is nothing we can do about it, header contents could
only be checked against chain state, there is nothing to check for internal
consistency.
NewMerkleTree is a memory hog, we can do better than that:
BenchmarkMerkle/NewMerkleTree-8 13 88434670 ns/op 20828207 B/op 300035 allocs/op
BenchmarkMerkle/CalcMerkleRoot-8 15 69264150 ns/op 0 B/op 0 allocs/op
Go-way of removing elements from slice is via `append` builtin.
There is a separate opcode for removing elements from
Arrays, which is cheaper and supported in this commit.
It's not needed, we're either creating a new one and assigning it 6 lines
above or we're changing already existing big.Int via a pointer, so no update
is needed.
We're at the point where even this code can clearly be seen in profiles. We
can save on some buffers (and CPU cycles) by encoding the answer once.
Another ~2% TPS for single node.
There is nothing requiring us to do so. It also is bad because it allows for
new transaction to replace some already existing one with the same fee
parameters just because it has "better" hash.
But the other thing is that for transactions with equal fees it's always
better for us to append them to the end of the list, instead of inserting them
in the middle, so this change allows to reduce slice item movements and gain
some 6-7% increase for single-node TPS.
Time is not really relevant for us here and we don't use this timestamp in any
way. Yet it occupies 24 bytes and we do two clock_gettime calls to get it.
Replace it with blockStamp which is going to be used in the future for
transaction retransmissions.
It allows to improve single-node TPS by another 3%.
We're constantly checking for transactions there and most of the time this
check is not successful (meaning that the transaction in question is
new). Bloom filter easily reduces the need to search over the DB in 99% of
these cases and gives some 13% increase in single-node TPS.
The cost of Y calculation from X is comparable with signature check, so it
reduces witness check overhead by ~30% for cached keys and gives ~5% overall
boost in TPS.
Most often we only need to read them and it doesn't require copying. Make an
explicit copy (and copy only things we need!) where needed.
After the recent neo-vm tests update our vm package testing time jumped to
~12s, with this change it's now more like ~8s.
Inspired by neo-project/neo-vm#352. We can't directly compare slices, so we're
better optimize things we already have. At the same time this code would
behave a bit different if A is to call B and then B is call A and then some
pointer from the first A invocation is to be compared with a pointer from the
second A invocation. Not sure it really matters.
Now we have VerifyTx() and PoolTx() APIs that either verify transaction in
isolation or verify it against the mempool (either the primary one or the one
given) and then add it there. There is no possibility to check against the
mempool, but not add a transaction to it, but I doubt we really need it.
It allows to remove some duplication between old PoolTx and verifyTx where
they both tried to check transaction against mempool (verifying first and then
adding it). It also saves us utility token balance check because it's done by
the mempool anyway and we no longer need to do that explicitly in verifyTx.
It makes AddBlock() and verifyBlock() transaction's checks more correct,
because previously they could miss that even though sender S has enough
balance to pay for A, B or C, he can't pay for all of them.
Caveats:
* consensus is running concurrently to other processes, so things could
change while verifyBlock() is iterating over transactions, this will be
mitigated in subsequent commits
Improves TPS value for single node by at least 11%.
Fixes#667, fixes#668.
It's cached in dbft for a view anyway, so there is no big difference here
from security POV. Lets us squeeze yet another 4% TPS improvement.
Make the system fail if unable to decrypt the key along the way, which is a
part of #1312.
New transactions are added to the chain with blocks. If there is no
transaction X at height N in DAO, it could only be added with block N+1, so
it has to be present there. Therefore we can replace `dao.HasTransaction()`
check with a search through in-block transactions. HasTransaction() is nasty
in that it may add useless load the DB and this code is being run with a big
Blockchain lock held, so we don't want to be delayed here at all.
Improves single-node TPS by ~2%.
The end effect is almost as if `VerifyTransactions: false` was set in the
config, but without actually compromising the guarantees provided by it.
It almost doubles performance for single-mode benchmarks and makes block
processing smoother (more smaller blocks are being produced).
