Commit graph

134 commits

Author SHA1 Message Date
Roman Khimov
c12a3b71d4 dao: drop GetStorageItems* APIs
They're just adding another useless caching layer to the Seek.
2022-03-31 19:18:51 +03:00
Anna Shaleva
9b841b9b8f core, rpc: use Seek to iterate over NEP* transfers
The results are controversial a bit. MemoryPS is a special storage
and the new approach doesn't help it to iterate through NEP17 transfers.
However, on long distances (if transfers are requested for more than
1000 blocks, which is ~4.1 hours of mainnet chain) both LevelDB and
BoltDB perform good enough with the new approach. Taking into account
the fact that default value for the query period of `getnep17transfers`
RPC handler is a week (~40K mainnet blocks) we can sacrifice MemoryPS
performance in favour of BoltDB and LevelDB optimisation.

Close #2263.

Benchmark results:

name                                                                           old time/op    new time/op    delta
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1_Take100Blocks-8           783µs ±13%    1762µs ± 4%  +125.12%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1_Take1000Blocks-8         6.91ms ± 2%    9.00ms ± 2%   +30.28%  (p=0.000 n=10+9)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-100_Take100Blocks-8        1.43ms ± 8%    1.79ms ± 4%   +24.93%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-100_Take1000Blocks-8       7.78ms ± 3%    8.93ms ± 2%   +14.78%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1000_Take100Blocks-8       7.69ms ± 3%    1.73ms ±10%   -77.50%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1000_Take1000Blocks-8      14.1ms ± 2%     9.0ms ± 2%   -36.53%  (p=0.000 n=9+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1_Take100Blocks-8          768µs ± 3%     801µs ± 2%    +4.24%  (p=0.000 n=10+9)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1_Take1000Blocks-8        8.03ms ± 2%    8.05ms ± 8%      ~     (p=0.436 n=10+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-100_Take100Blocks-8       1.70ms ±16%    0.85ms ± 7%   -49.85%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-100_Take1000Blocks-8      10.8ms ± 2%     8.1ms ± 2%   -25.21%  (p=0.000 n=10+8)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1000_Take100Blocks-8      10.8ms ± 2%     0.9ms ± 7%   -92.12%  (p=0.000 n=9+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1000_Take1000Blocks-8     18.2ms ±13%     8.2ms ± 6%   -54.95%  (p=0.000 n=10+9)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1_Take100Blocks-8         874µs ± 6%     823µs ± 2%    -5.81%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1_Take1000Blocks-8       9.35ms ± 2%    8.14ms ± 2%   -12.91%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-100_Take100Blocks-8      1.85ms ± 4%    0.89ms ±10%   -51.68%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-100_Take1000Blocks-8     10.6ms ± 4%     8.2ms ± 2%   -22.44%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1000_Take100Blocks-8     10.7ms ± 2%     0.9ms ± 2%   -91.90%  (p=0.000 n=8+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1000_Take1000Blocks-8    21.8ms ±15%     8.2ms ± 1%   -62.25%  (p=0.000 n=10+10)

name                                                                           old alloc/op   new alloc/op   delta
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1_Take100Blocks-8           650kB ± 0%    1010kB ± 7%   +55.45%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1_Take1000Blocks-8         6.39MB ± 0%    9.89MB ± 1%   +54.74%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-100_Take100Blocks-8        1.28MB ± 0%    1.14MB ± 0%   -11.25%  (p=0.000 n=9+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-100_Take1000Blocks-8       7.02MB ± 0%   10.01MB ± 0%   +42.54%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1000_Take100Blocks-8       7.02MB ± 0%    1.08MB ± 0%   -84.64%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1000_Take1000Blocks-8      12.8MB ± 0%    10.0MB ± 0%   -22.04%  (p=0.000 n=10+9)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1_Take100Blocks-8          823kB ± 3%     866kB ± 5%    +5.19%  (p=0.001 n=9+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1_Take1000Blocks-8        9.92MB ± 0%    9.85MB ± 0%    -0.71%  (p=0.000 n=10+9)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-100_Take100Blocks-8       1.84MB ± 4%    1.01MB ± 5%   -44.75%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-100_Take1000Blocks-8      11.0MB ± 0%    10.0MB ± 1%    -8.45%  (p=0.000 n=10+9)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1000_Take100Blocks-8      11.0MB ± 0%     1.0MB ± 1%   -90.55%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1000_Take1000Blocks-8     20.0MB ± 0%    10.1MB ± 0%   -49.69%  (p=0.000 n=9+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1_Take100Blocks-8         913kB ± 5%     907kB ± 6%      ~     (p=0.497 n=9+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1_Take1000Blocks-8       10.0MB ± 1%    10.0MB ± 1%    -0.63%  (p=0.001 n=10+8)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-100_Take100Blocks-8      1.92MB ± 2%    1.04MB ± 0%   -45.53%  (p=0.000 n=9+8)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-100_Take1000Blocks-8     11.1MB ± 1%    10.1MB ± 0%    -9.22%  (p=0.000 n=10+7)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1000_Take100Blocks-8     11.1MB ± 1%     1.0MB ± 0%   -90.61%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1000_Take1000Blocks-8    20.4MB ± 1%    10.1MB ± 0%   -50.46%  (p=0.000 n=10+10)

name                                                                           old allocs/op  new allocs/op  delta
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1_Take100Blocks-8           11.1k ± 0%     11.7k ± 0%    +5.35%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1_Take1000Blocks-8           110k ± 0%      110k ± 0%    +0.55%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-100_Take100Blocks-8         22.0k ± 0%     11.9k ± 0%   -46.14%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-100_Take1000Blocks-8         120k ± 0%      110k ± 0%    -8.44%  (p=0.000 n=10+9)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1000_Take100Blocks-8         120k ± 0%       12k ± 0%   -90.36%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/MemPS_StartFromBlockN-1000_Take1000Blocks-8        219k ± 0%      110k ± 0%   -49.73%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1_Take100Blocks-8          11.3k ± 0%     11.2k ± 0%    -1.06%  (p=0.000 n=9+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1_Take1000Blocks-8          112k ± 0%      110k ± 0%    -2.36%  (p=0.000 n=9+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-100_Take100Blocks-8        22.5k ± 0%     11.3k ± 0%   -49.65%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-100_Take1000Blocks-8        124k ± 0%      110k ± 0%   -11.08%  (p=0.000 n=9+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1000_Take100Blocks-8        124k ± 0%       11k ± 0%   -90.84%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/BoltPS_StartFromBlockN-1000_Take1000Blocks-8       225k ± 0%      110k ± 0%   -51.07%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1_Take100Blocks-8         11.4k ± 0%     11.3k ± 0%    -1.36%  (p=0.000 n=9+9)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1_Take1000Blocks-8         114k ± 0%      111k ± 0%    -2.33%  (p=0.000 n=10+9)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-100_Take100Blocks-8       22.7k ± 0%     11.5k ± 0%   -49.56%  (p=0.000 n=9+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-100_Take1000Blocks-8       125k ± 0%      111k ± 0%   -11.12%  (p=0.000 n=10+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1000_Take100Blocks-8       125k ± 0%       11k ± 0%   -90.82%  (p=0.000 n=8+10)
Blockchain_ForEachNEP17Transfer/LevelPS_StartFromBlockN-1000_Take1000Blocks-8      229k ± 1%      111k ± 0%   -51.42%  (p=0.000 n=10+10)
2022-01-19 20:55:08 +03:00
Roman Khimov
ce9d0b22cf *: use NEP-XX naming consistently in docs/comments
Standards are NEP-11 and NEP-17, not NEP11, not NEP17, not anything
else. Variable/function names of course can use whatever fits, but documents
and comments should be consistent wrt this.
2021-11-19 12:58:46 +03:00
Roman Khimov
7f40a0cfd8 state: prime token transfer info with the demand for new logs
Setting NewNEPXXBatch avoids the need to get them from the database which is
useful for newly tracked accounts.
2021-11-16 22:52:10 +03:00
Roman Khimov
c63aeb38bb state: prepare for NEP-11 data tracking, refactor/rename
There is a lot of similarity, so try reusing common code and use more neutral
naming.
2021-11-16 19:18:06 +03:00
Anna Shaleva
72726d46d3 core: refactor callers of MemCachedStore.Seek
MemCachedStore.Seek now sorts results, so its callers may omit sorting.
2021-10-21 10:05:12 +03:00
Evgeniy Stratonikov
23adb1e2fc state: optimize NEP17TransferLog.Append
Do not allocate a separate buffer for the transfer.
```
name                       old time/op    new time/op    delta
NEP17TransferLog_Append-8    58.8µs ± 3%    32.1µs ± 1%  -45.40%  (p=0.000 n=10+9)

name                       old alloc/op   new alloc/op   delta
NEP17TransferLog_Append-8     118kB ± 1%      44kB ± 3%  -63.00%  (p=0.000 n=9+10)

name                       old allocs/op  new allocs/op  delta
NEP17TransferLog_Append-8       901 ± 1%       513 ± 3%  -43.08%  (p=0.000 n=9+8)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 11:09:49 +03:00
Evgeniy Stratonikov
403a4b75de state/test: add benchmark for NEP17TransferLog.Append
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 11:09:49 +03:00
Evgeniy Stratonikov
b210a34b1e state: optimize NEP17Balance deserialization
```
BenchmarkNEP17BalanceFromBytes/stackitem-8         	 2402318	       503.3 ns/op	     208 B/op	      10 allocs/op
BenchmarkNEP17BalanceFromBytes/from_bytes-8        	 7623139	       160.7 ns/op	      72 B/op	       3 allocs/op
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 11:09:49 +03:00
Evgeniy Stratonikov
3218b74ea5 state: optimize NEP17Balance serialization
Put to slice directly and allow to provide pre-allocated buffer.
```
BenchmarkNEP17BalanceBytes/stackitem-8         	 1712475	       673.4 ns/op	     448 B/op	       9 allocs/op
BenchmarkNEP17BalanceBytes/bytes-8             	13422715	        75.80 ns/op	      32 B/op	       2 allocs/op
BenchmarkNEP17BalanceBytes/bytes,_prealloc-8   	25990371	        46.46 ns/op	      16 B/op	       1 allocs/op
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 11:09:06 +03:00
Evgeniy Stratonikov
291a29af1e *: do not use WriteArray for frequently used items
`WriteArray` involves reflection, it makes sense to optimize
serialization of transactions and application logs which are serialized
constantly. Adding case in a type switch in `WriteArray` is not an
option because we don't want new dependencies for `io` package.

```
name                          old time/op    new time/op    delta
AppExecResult_EncodeBinary-8     852ns ± 3%     656ns ± 2%  -22.94%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
AppExecResult_EncodeBinary-8      448B ± 0%      376B ± 0%  -16.07%  (p=0.000 n=10+10)

name                          old allocs/op  new allocs/op  delta
AppExecResult_EncodeBinary-8      7.00 ± 0%      5.00 ± 0%  -28.57%  (p=0.000 n=10+10)
```

```
name                 old time/op    new time/op    delta
Transaction_Bytes-8    1.29µs ± 3%    0.76µs ± 5%  -41.52%  (p=0.000 n=9+10)

name                 old alloc/op   new alloc/op   delta
Transaction_Bytes-8    1.21kB ± 0%    1.01kB ± 0%  -16.56%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
Transaction_Bytes-8      12.0 ± 0%       7.0 ± 0%  -41.67%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-06 11:59:20 +03:00
Roman Khimov
dede4fa7b1 state: convert NEO balance to stack item directly
Avoid calling Append() that will reallocate the slice, we know the length of
the slice exactly.
2021-08-03 17:59:38 +03:00
Anna Shaleva
e8bed184d5 core: implement dynamic NEP17 balances tracking
Request NEP17 balances from a set of NEP17 contracts instead of getting
them from storage. LastUpdatedBlock tracking remains untouched, because
there's no way to retrieve it dynamically.
2021-07-29 10:23:01 +03:00
Anna Shaleva
e46d76d7aa core: rename state.NEP17Balances to state.NEP17TransferInfo
Balances are to be removed from state.NEP17TransferInfo, so the remnant
fields are NextTransferBatch, NewBatch and a map of LastUpdatedBlocks.
These fields are more staff-related.

Also rename dao.[Get, Put, put]NEP17Balances and STNEP17Balances
preffix.

Also rename NEP17TransferInfo.Trackers to LastUpdatedBlockTrackers
because NEP17TransferInfo.Balances are to be removed.
2021-07-28 13:22:53 +03:00
Roman Khimov
588f3fbbd3 state: drop State from NEOBalance and NEP17Balance
We're in the `state` package already.
2021-07-19 22:01:07 +03:00
Roman Khimov
5f13de3a76 *: simplify some integer checks with IsUint64()
If we need a positive number we can do `IsUint64()` instead of checking that
`Int64()` is `> 0`.
2021-07-19 15:42:42 +03:00
Roman Khimov
233307aca5 stackitem: completely drop MaxArraySize
Turns out C# VM doesn't have it since preview2, so our limiting of
MaxArraySize in incompatible with it. Removing this limit shouldn't be a
problem with the reference counter we have, both APPEND and SETITEM add things
to reference counter and we can't exceed MaxStackSize. PACK on the other hand
can't get more than MaxStackSize-1 of input elements.

Unify NEWSTRUCT with NEWARRAY* and use better integer checks at the same time.

Multisig limit is still 1024.
2021-07-19 15:42:42 +03:00
Roman Khimov
aab18c3083 stackitem: introduce Convertible interface
We have a lot of native contract types that are converted to stack items
before serialization, then deserialized as stack items and converted back to
regular structures. stackitem.Convertible allows to remove a lot of repetitive
io.Serializable code.

This also introduces to/from converter in testserdes which unfortunately
required to change util tests to avoid circular references.
2021-07-19 15:42:42 +03:00
Roman Khimov
2d993d0da5 state: store notary deposit as stackitem
Which is less effective, but makes it more similar to other native contracts
that are supposed to be contracts anyway.
2021-07-19 15:42:42 +03:00
Roman Khimov
0cd9cd0c80 state/result: save/return real JSONization errors
Don't hide/obfuscate real problems.
2021-07-07 00:42:36 +03:00
Roman Khimov
e62a766058 state: move nil check down to stackitem JSON processing
We want to return real errors, not some generic thing for any kind of thing
happening.
2021-07-07 00:38:19 +03:00
Roman Khimov
0de949b575 stackitem: remove Item/StackItem from function names
They're useless in a package named 'stackitem', make this package a bit more
user-friendly.
2021-07-06 19:56:23 +03:00
Anna Shaleva
999fba81cd core: export (*NEOBalanceState).FromStackItem 2021-05-31 11:18:54 +03:00
Anna Shaleva
c9099fa555 core: remove native nonfungible token 2021-05-17 22:22:07 +03:00
Roman Khimov
9d2712573f *: enable godot linter and fix all its warnings
It's important for NeoGo to have clean documentation. No functional changes.
2021-05-12 23:17:03 +03:00
Roman Khimov
c4e084b0d8 *: fix whitespace errors
leading/trailing newlines
2021-05-12 22:51:41 +03:00
Roman Khimov
5f7ec6e784 state: fix MPTRoot JSONization
Match C# implementation. It's easier to do this by changing the structure
itself.
2021-03-26 13:45:18 +03:00
Roman Khimov
8c110a6147 state: drop Network from MPTRoot structure
It's only needed to verify/sign.
2021-03-26 13:45:18 +03:00
Roman Khimov
0248e2c0d2 state: add network to MPTRoot, change hashing
Fix RC1 incompatibility.
2021-03-25 22:37:30 +03:00
Anna Shaleva
9377751e65 config: add NativeUpdateHistory 2021-03-15 12:35:14 +03:00
Evgeniy Stratonikov
bf20db09e0 stateroot: move state-root related logic to core/stateroot 2021-03-09 13:48:29 +03:00
Evgeniy Stratonikov
e551432b30 dao: serialize state.StorageItem as raw bytes 2021-03-09 12:11:26 +03:00
Evgeniy Stratonikov
55698d0426 dao: use raw state.StorageItem instead of pointer
It is now a slice, there is no need for additional indirection.
2021-03-09 12:11:25 +03:00
Evgeniy Stratonikov
fd4174ad31 core/state: remove IsConst from StorageItem 2021-03-09 11:59:08 +03:00
Roman Khimov
e6d3a60431
Merge pull request #1782 from nspcc-dev/optimize/nep17
dao: do not look up new NEP17 batch
2021-03-02 15:28:42 +03:00
Evgeniy Stratonikov
d5cb95e685 dao: do not look up new NEP17 batch
It will fail anyway.
2021-02-26 14:08:51 +03:00
Evgeniy Stratonikov
f9f1fe03b2 core: refactor native call
1. `System.Contract.CallNative` expects version on stack.
2. Actual method is determined based on current
   instruction pointer.
3. Native hashes don't longer depend on NEF checksum.
2021-02-26 10:59:09 +03:00
Anna Shaleva
1e108f20a7 core: remove description from NFT state 2021-02-17 13:19:26 +03:00
Evgeniy Stratonikov
f0d8652bcd rpc: implement getnativecontracts RPC 2021-02-11 12:27:43 +03:00
Evgeniy Stratonikov
e1d2a5b5b7 state: split ContractBase and UpdateCounter
Latter doesn't make sense for native contracts.
2021-02-11 12:24:09 +03:00
Evgeniy Stratonikov
b0fbd897ad Revert "rpc: marshal fees and GAS as Fixed8 decimal"
This reverts commit a79b12b4d4.
2021-02-09 11:16:52 +03:00
Anna Shaleva
a2715404d0 core: serialise manifest as stackitem 2021-02-05 19:10:15 +03:00
Evgeniy Stratonikov
e4ff8326b5 native: add NameService 2021-02-01 21:40:21 +03:00
Roman Khimov
054ca27e9c state: use checksums and names to calculate contract hashes
It allows to deploy the same NEF using one sender and get different contract
hashes. See neo-project/neo#2240.
2021-01-22 12:22:48 +03:00
Evgeniy Stratonikov
719dceff77 nef: merge Compiler and Version fields 2021-01-19 11:19:28 +03:00
Evgeniy Stratonikov
52843fc1bf nef: add Tokens field 2021-01-19 11:19:24 +03:00
Roman Khimov
6ecc6f0422 native: call native contracts by ID instead of name
Fix #1666.
2021-01-18 00:38:23 +03:00
Roman Khimov
f39ede9869 opcode: add CALLT opcode, move ABORT/ASSERT
Refs. #1644. Hash compatibility test temporarily disabled, to be enabled when
it's up to date with current C# master.
2021-01-18 00:14:52 +03:00
Evgeniy Stratonikov
0b26b46234 state: store NEF instead of script for contract
NEFs for native contracts are set statically, thus
field values are taken from the reference implementation.
2021-01-13 15:34:10 +03:00
Evgenii Stratonikov
8c22d27acc state: allow to encode AppExecResult with recursive items
1. Encode them to a special type, decode to `nil`.
2. `Interop` can be encoded in JSON, this info should also be preserved.
2020-12-18 13:04:31 +03:00