Commit graph

516 commits

Author SHA1 Message Date
Roman Khimov
a68a8aa8fc core: simplify and correct notification handling
* both 'to' and 'from' are either Null or Hash160, there is no other
   possibility for valid NEP-17. So returning util.Uint160{} in case of
   parsing error is wrong.
 * but this is what allowed burns/mints to work at the expense of error
   allocation inside of util.Uint160DecodeBytesBE()
 * Uint160 can technically fit into regular VM integer, so even though it'd be
   quite surprising to see it there, TryBytes() is more correct (and easier!)
   to use
 * same thing with `amount`, we have `TryInteger()` that easily covers all
   possible cases and does appropriate error checking inside
2021-08-20 11:26:16 +03:00
Anna Shaleva
5f9d38f640 core: refactor (*DAO).StoreAsTransaction
Squash (*DAO).StoreAsTransaction and
(*DAO).StoreConflictingTransactions. It's better to keep them this way,
because StoreAsTransaction is always followed by
StoreConflictingTransactions, so it's an atomic operation.

The logic wasn't changed.
2021-08-18 13:39:28 +03:00
Anna Shaleva
4b35a1cf92 core: remove conflicting transactions wrt MaxTraceableBlocks 2021-08-18 13:31:47 +03:00
Roman Khimov
adc660c3e0
Merge pull request #2123 from nspcc-dev/store-better
Store better
2021-08-13 12:50:24 +03:00
Roman Khimov
47f0f4c45f dao: completely drop Cached
It was very useful in 2.0 days, but today it only serves one purpose that
could easily (and more effectively!) be solved in another way.
2021-08-11 23:06:17 +03:00
Roman Khimov
3e60771175 core: deduplicate and simplify processNEP17Transfer a bit
Just refactoring, no functional changes.
2021-08-11 22:36:26 +03:00
Anna Shaleva
0e3b9c48a2 core: add API to store StateSyncPoint and StateSyncCurrentBlockHeight
We need it in order not to mess up the blockchain which has its own
CurrentBlockHeight.
2021-08-10 14:06:28 +03:00
Anna Shaleva
cb01f533c0 core: store conflicting transactions in a separate method
(DAO).StoreConflictingTransactions will be reused from the state sync
module.
2021-08-10 13:47:13 +03:00
Anna Shaleva
35501a281a core: remove untraceable blocks wrt StateSyncInterval 2021-08-10 13:47:10 +03:00
Anna Shaleva
76c687aaa1 config: add P2PStateExchangeExtensions and StateSyncInterval settings 2021-08-10 11:00:32 +03:00
Roman Khimov
39f874d03f core: don't recalculate witness script hash
We know it already, but with current loading code VM will hash it once
more. It doesn't help a lot and still it costs nothing to avoid this
overhead.

name             old time/op    new time/op    delta
VerifyWitness-8    93.4µs ± 3%    92.7µs ± 2%    ~     (p=0.353 n=10+10)

name             old alloc/op   new alloc/op   delta
VerifyWitness-8    3.43kB ± 0%    3.40kB ± 0%  -0.70%  (p=0.000 n=9+9)

name             old allocs/op  new allocs/op  delta
VerifyWitness-8      67.0 ± 0%      66.0 ± 0%  -1.49%  (p=0.000 n=10+10)
2021-08-06 11:25:09 +03:00
Roman Khimov
f8174ca64c core: ensure data logged is from persistent store
Using bc.dao here is wrong, it can contain unpersisted data.
2021-08-02 16:33:09 +03:00
Roman Khimov
8277b7a19a core: don't spawn goroutine for persist function
It doesn't make any sense, in some situations it leads to a number of
goroutines created that will Persist one after another (as we can't Persist
concurrently). We can manage it better in a single thread.

This doesn't change performance in any way, but somewhat reduces resource
consumption. It was tested neo-bench (single node, 10 workers, LevelDB) on two
machines and block dump processing (RC4 testnet up to 62800 with VerifyBlocks
set to false) on i7-8565U.

Reference (b9be892bf9):

Ryzen 9 5950X:
RPS     27747.349 27407.726 27520.210  ≈ 27558   ± 0.63%
TPS     26992.010 26993.468 27010.966  ≈ 26999   ± 0.04%
CPU %      28.928    28.096    29.105  ≈    28.7 ± 1.88%
Mem MB    760.385   726.320   756.118  ≈   748   ± 2.48%

Core i7-8565U:
RPS     7783.229 7628.409 7542.340  ≈ 7651   ± 1.60%
TPS     7708.436 7607.397 7489.459  ≈ 7602   ± 1.44%
CPU %     74.899   71.020   72.697  ≈   72.9 ± 2.67%
Mem MB   438.047  436.967  416.350  ≈  430   ± 2.84%

DB restore:
real    0m20.838s 0m21.895s 0m21.794s  ≈ 21.51 ± 2.71%
user    0m39.091s 0m40.565s 0m41.493s  ≈ 40.38 ± 3.00%
sys      0m3.184s  0m2.923s  0m3.062s  ≈  3.06 ± 4.27%

Patched:

Ryzen 9 5950X:
RPS     27636.957 27246.911 27462.036  ≈ 27449   ±  0.71%  ↓ 0.40%
TPS     27003.672 26993.468 27011.696  ≈ 27003   ±  0.03%  ↑ 0.01%
CPU %      28.562    28.475    28.012  ≈    28.3 ±  1.04%  ↓ 1.39%
Mem MB    627.007   648.110   794.895  ≈   690   ± 13.25%  ↓ 7.75%

Core i7-8565U:
RPS     7497.210 7527.797 7897.532  ≈ 7641   ±  2.92%  ↓ 0.13%
TPS     7461.128 7482.678 7841.723  ≈ 7595   ±  2.81%  ↓ 0.09%
CPU %     71.559   73.423   69.005  ≈   71.3 ±  3.11%  ↓ 2.19%
Mem MB   393.090  395.899  482.264  ≈  424   ± 11.96%  ↓ 1.40%

DB restore:
real    0m20.773s 0m21.583s 0m20.522s  ≈ 20.96 ±  2.65%  ↓ 2.56%
user    0m39.322s 0m42.268s 0m38.626s  ≈ 40.07 ±  4.82%  ↓ 0.77%
sys      0m3.006s  0m3.597s  0m3.042s  ≈  3.22 ± 10.31%  ↑ 5.23%
2021-08-02 16:33:00 +03:00
Roman Khimov
49be753850 core: spread storeBlock actions to three goroutines
Block processing consists of:
 * saving block/transactions to the DB
 * executing blocks/transactions
 * processing notifications/saving AERs
 * updating MPT
 * atomically updating Blockchain state

Of these the first one is completely independent of others, it can be done in
a separate routine easily. The third one technically depends on the second,
it just doesn't have data until something is executed. At the same time it
doesn't affect future executions in any way, so we can offload
AER/notification processing to separate goroutine (while the main thread
proceeds with other transactions).

MPT update depends on all executions, so it can't be offloaded, but it can be
done concurrently to AER processing. And only the last thing actually needs
all previous ones to be finished, so it's a natural synchronization point.

So we spawn two additional routines and let the main one execute transactions
and update MPT as fast as it can. While technically all of these routines
could share single DAO (they are working with different KV sets) benchmarking
shows that using separate DAOs and then persisting them to lower one actually
works about 7-8%% better. At the same time we can simplify DAOs used, Cached
one is only relevant for AER processing because it caches NEP-17 tracking
data, everything else can do just fine with Simple.

The change was tested for performance with neo-bench (single node, 10 workers,
LevelDB) on two machines and block dump processing (RC4 testnet up to 50825
with VerifyBlocks set to false) on i7-8565U. neo-bench creates huge blocks
with lots of transactions while RC4 dump mostly consists of empty blocks.

Reference results (06c3dda5d1):

Ryzen 9 5950X:
RPS ≈ 20059.569   21186.328   20158.983   ≈ 20468   ±  3.05%
TPS ≈ 19544.993   20585.450   19658.338   ≈ 19930   ±  2.86%
CPU ≈    18.682%     23.877%     22.852%  ≈    21.8 ± 12.62%
Mem ≈   618.981MB   559.246MB   541.539MB ≈   573   ±  7.08%

Core i7-8565U:
RPS ≈ 5927.082   6526.739   6372.115   ≈ 6275   ± 4.96%
TPS ≈ 5899.531   6477.187   6329.515   ≈ 6235   ± 4.81%
CPU ≈   56.346%    61.955%    58.125%  ≈   58.8 ± 4.87%
Mem ≈  212.191MB  224.974MB  205.479MB ≈  214   ± 4.62%

DB restore:
real    0m12.683s 0m13.222s 0m13.382s  ≈ 13.096 ±  2.80%
user    0m18.501s 0m19.163s 0m19.489s  ≈ 19.051 ±  2.64%
sys      0m1.404s  0m1.396s  0m1.666s  ≈  1.489 ± 10.32%

After the change:

Ryzen 9 5950X:
RPS ≈ 23056.899   22822.015   23006.543   ≈ 22962   ± 0.54%
TPS ≈ 22594.785   22292.071   22800.857   ≈ 22562   ± 1.13%
CPU ≈    24.262%     23.185%     25.921%  ≈    24.5 ± 5.65%
Mem ≈   614.254MB   613.204MB   555.491MB ≈   594   ± 5.66%

Core i7-8565U:
RPS ≈ 6378.702   6423.927   6363.788      ≈ 6389   ± 0.49%
TPS ≈ 6327.072   6372.552   6311.179      ≈ 6337   ± 0.50%
CPU ≈   57.599%    58.622%    59.737%     ≈   58.7 ± 1.82%
Mem ≈  198.697MB  188.746MB  200.235MB    ≈  196   ± 3.18%

DB restore:
real    0m13.576s 0m13.334s 0m12.757s  ≈  13.222 ±  3.18%
user    0m19.113s 0m19.490s 0m20.197s  ≈  19.600 ±  2.81%
sys      0m2.211s  0m1.558s  0m1.559s  ≈   1.776 ± 21.21%

On Ryzen 9 we've got 12% better RPS, 13% better TPS with 12% CPU and 3% RAM
more used. Core i7-8565U changes don't seem to be statistically significant:
1.8% more RPS, 1.6% more TPS with about the same CPU and 8.5% less RAM
used. It also is 1% worse in DB restore time.

The result is somewhat expected, on a powerful machine with lots of spare
cores we get 10%+ better results while on average resource-constrained laptop it
doesn't change much (the machine is already saturated). Overall, this seems to
be worthwhile.
2021-07-30 15:45:17 +03:00
Anna Shaleva
a30e48ff90 core: increment the DB version
DB scheme has been changed.
2021-07-29 10:23:13 +03:00
Anna Shaleva
e8bed184d5 core: implement dynamic NEP17 balances tracking
Request NEP17 balances from a set of NEP17 contracts instead of getting
them from storage. LastUpdatedBlock tracking remains untouched, because
there's no way to retrieve it dynamically.
2021-07-29 10:23:01 +03:00
Anna Shaleva
e46d76d7aa core: rename state.NEP17Balances to state.NEP17TransferInfo
Balances are to be removed from state.NEP17TransferInfo, so the remnant
fields are NextTransferBatch, NewBatch and a map of LastUpdatedBlocks.
These fields are more staff-related.

Also rename dao.[Get, Put, put]NEP17Balances and STNEP17Balances
preffix.

Also rename NEP17TransferInfo.Trackers to LastUpdatedBlockTrackers
because NEP17TransferInfo.Balances are to be removed.
2021-07-28 13:22:53 +03:00
Roman Khimov
36d486a664 config: add InitialGASSupply, fix #2073
We now have 52M by default.
2021-07-20 16:59:54 +03:00
Roman Khimov
f4a9139a05
Merge pull request #2071 from nspcc-dev/fix-more-vm-bugs
Fix more VM bugs
2021-07-19 22:00:17 +03:00
Roman Khimov
aab18c3083 stackitem: introduce Convertible interface
We have a lot of native contract types that are converted to stack items
before serialization, then deserialized as stack items and converted back to
regular structures. stackitem.Convertible allows to remove a lot of repetitive
io.Serializable code.

This also introduces to/from converter in testserdes which unfortunately
required to change util tests to avoid circular references.
2021-07-19 15:42:42 +03:00
Roman Khimov
21e05f5779
Merge pull request #2055 from nspcc-dev/fix-gas-limits
Tune fixed GAS limits
2021-07-16 16:53:06 +03:00
Evgeniy Stratonikov
8077f9232d interop: implement System.Runtime.GetRandom
Our murmur3 implementation is architecture independent and optimized in
assembly.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-15 16:00:01 +03:00
Evgeniy Stratonikov
451b02122a *: increase GAS for verification
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-14 10:27:09 +03:00
Roman Khimov
1853d0c713 core: don't copy config in ApplyPolicyToTxSet
Makes no sense, GetConfig() is for external users.
2021-07-09 13:01:42 +03:00
Anna Shaleva
9673a04009 core: allow to add several headers with StateRootInHeader on
Problem: with StateRootInHeader setting on only one header of height N+1
can be added to the chain of height N, because we need local stateroot
to verify headers (which is calculated for the last stored block N).
Thus, adding chunk of headers starting from the current chain's heigh
is impossible and (*Blockchain).AddHeaders doesn't have much sense.

Solution: verify header.PrevStateRoot only for header N+1. Rest of the
headers should be added without PrevStateRoot verification.
2021-06-30 11:56:05 +03:00
Anna Shaleva
6ff64153db core: check stored native states against autogenerated ones on start 2021-06-10 22:16:05 +03:00
Evgeniy Stratonikov
33e1e61343 config: make MaxValidUntilBlockIncrement configurable 2021-05-17 13:43:03 +03:00
Roman Khimov
c4e084b0d8 *: fix whitespace errors
leading/trailing newlines
2021-05-12 22:51:41 +03:00
Roman Khimov
75a55d910e
Merge pull request #1956 from nspcc-dev/interop/checksig-rename
core: rename Neo.Crypto.[CheckSig CheckMultisig] interops
2021-05-12 17:06:39 +03:00
Anna Shaleva
4b933f88a7 core: simplify interop functions
We now have the only interop table (system interops).
2021-05-12 13:30:01 +03:00
Evgeniy Stratonikov
fbf7d9edce core: drop oracles from extensible whitelist 2021-05-11 12:16:24 +03:00
Roman Khimov
508ad51a86 core: ignore non-numeric "amounts" in Transfer events
Refs. #1900.
2021-04-12 15:09:37 +03:00
Roman Khimov
1785a333d2 core: tune "blockchain persist completed" message
It's misleading, we're writing it on disk-level persistence, it's not the same
as block "persistence" in C#, so we better write it clear.
2021-04-07 18:14:30 +03:00
Roman Khimov
9b15ab48ad core: restore MPT collapsing logic
Fix #1867, revert part of bf20db09e0.
2021-04-07 11:14:18 +03:00
Roman Khimov
d143696328 stateroot: try to fix MPT caching/updating
It was completely ruined by bf20db09e0. MPT was
updating bc.dao directly which shouldn't ever happen, it must write into the
same cache and then Persist these KVs as usual.
2021-03-27 00:33:11 +03:00
Roman Khimov
56fd375c6d core: move stateroot check into header check
As it's a part of the header.
2021-03-27 00:33:11 +03:00
Roman Khimov
b3f9cd1541 dao: drop network from DAO
Not used any more.
2021-03-26 13:45:18 +03:00
Roman Khimov
df12adaa9e crypto: remove crypto.Verifiable interface
We can now verify any hash.Hashable thing.
2021-03-26 13:45:18 +03:00
Anna Shaleva
681bd89cc7 core: move native node roles to a separate pkg
We need this to avoid `native` dependency in the RPC client.
2021-03-23 13:56:34 +03:00
Roman Khimov
42465dd002
Merge pull request #1825 from nspcc-dev/fix-invokecontractverify
rpc: refactor invokecontractverify
2021-03-17 18:08:23 +03:00
Roman Khimov
d1251b8daf core: don't limit on/postPersist methods
They read/write/call/notify, so they need everything.
2021-03-16 22:48:32 +03:00
Roman Khimov
025330f132
Merge pull request #1832 from nspcc-dev/consensus/microfixes
consensus: add MaxBlockSize and MaxBlockSysFee settings
2021-03-16 18:11:33 +03:00
Anna Shaleva
20b70ee9fe config: add MaxBlockSystemFee setting 2021-03-16 12:09:11 +03:00
Anna Shaleva
8f06bf21d7 config: add MaxBlockSize setting 2021-03-16 12:08:47 +03:00
Anna Shaleva
9377751e65 config: add NativeUpdateHistory 2021-03-15 12:35:14 +03:00
Anna Shaleva
1261dd5306 core: refactor initVerificationVM
It needs only VM and function to get contract state. Also exposed this
method and extended Blockchainer. These changes are needed for the
next commit.
2021-03-12 18:55:27 +03:00
Evgeniy Stratonikov
ac95510402 core: do not return block if only header is present 2021-03-10 13:38:44 +03:00
Evgeniy Stratonikov
f83b376181 block: replace Base with Header 2021-03-10 13:38:44 +03:00
Evgeniy Stratonikov
ac227a80fe stateroot: use RoleStateValidator for verification 2021-03-09 13:51:10 +03:00
Evgeniy Stratonikov
bf20db09e0 stateroot: move state-root related logic to core/stateroot 2021-03-09 13:48:29 +03:00