Commit graph

587 commits

Author SHA1 Message Date
Roman Khimov
d638aee7a1
Merge pull request #2053 from nspcc-dev/stackitem-json
Speedup stackitem json serialization
2021-07-13 11:27:33 +03:00
Evgeniy Stratonikov
0611031fd4 stackitem/test: improve coverage for serialization
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-13 11:06:19 +03:00
Evgeniy Stratonikov
8fdc7e32f5 stackitem: cache serialized results in EncodeBinary
```
name            old time/op    new time/op    delta
EncodeBinary-8    8.39ms ± 1%    0.05ms ±21%  -99.44%  (p=0.000 n=10+9)

name            old alloc/op   new alloc/op   delta
EncodeBinary-8    2.24MB ± 0%    0.33MB ± 0%  -85.29%  (p=0.000 n=9+10)

name            old allocs/op  new allocs/op  delta
EncodeBinary-8     65.6k ± 0%      0.0k ± 0%  -99.95%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-13 11:06:19 +03:00
Evgeniy Stratonikov
17a3f17c74 stackitem: use byte-slice directly during encoding
```
name            old time/op    new time/op    delta
EncodeBinary-8    10.6ms ± 1%     8.4ms ± 1%  -20.94%  (p=0.000 n=9+10)

name            old alloc/op   new alloc/op   delta
EncodeBinary-8    1.64MB ± 0%    2.24MB ± 0%  +36.18%  (p=0.000 n=8+9)

name            old allocs/op  new allocs/op  delta
EncodeBinary-8      131k ± 0%       66k ± 0%  -49.98%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-13 11:06:18 +03:00
Evgeniy Stratonikov
c4eb7db8a5 stackitem: access value of Array/Struct directly
```
name      old time/op    new time/op    delta
ToJSON-8    50.3µs ±34%    47.8µs ± 9%     ~     (p=0.971 n=10+10)

name      old alloc/op   new alloc/op   delta
ToJSON-8     396kB ± 0%     396kB ± 0%   -0.10%  (p=0.000 n=6+10)

name      old allocs/op  new allocs/op  delta
ToJSON-8      34.0 ± 0%      18.0 ± 0%  -47.06%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:40:21 +03:00
Evgeniy Stratonikov
dc0a17dd0e stackitem: cache visited items while marshaling json
```
name      old time/op    new time/op    delta
ToJSON-8    4.52ms ± 4%    0.05ms ±34%  -98.89%  (p=0.000 n=8+10)

name      old alloc/op   new alloc/op   delta
ToJSON-8    2.13MB ± 0%    0.40MB ± 0%  -81.44%  (p=0.000 n=9+6)

name      old allocs/op  new allocs/op  delta
ToJSON-8     65.6k ± 0%      0.0k ± 0%  -99.95%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:40:21 +03:00
Evgeniy Stratonikov
69cdd5252a stackitem: add benchmark for serialization routines
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:40:17 +03:00
Roman Khimov
fe3c68b92d vm: panic in cloneIfStruct
It's internal, so we can deduplicate code a bit.
2021-07-12 11:52:24 +03:00
Roman Khimov
4000dd692c vm: make cloning limits a bit more effective
Take actual reference counter value into account.
2021-07-11 19:37:06 +03:00
Roman Khimov
cfe41abd35 vm: fix OOM during nested structure cloning
Resulting item can't have more than MaxStackSize elements. Technically this
limits to MaxStackSize cloned elements but it's considered to be enough to
mitigate the issue (the next size check is going to happen during push to the
stack). See neo-project/neo#2534, thanks @vang1ong7ang.
2021-07-11 18:47:50 +03:00
Evgeniy Stratonikov
1863023e27 vm: copy argument in unary opcodes, fix #2051
We already do this in binary opcodes, let's be consistent.

This fixes state difference at height 275663
for tx 3c498317684d63849b03e4c58ad57ce4b19bb206b7b01bcc64233de3b3e207f4

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-08 17:09:04 +03:00
Roman Khimov
e62a766058 state: move nil check down to stackitem JSON processing
We want to return real errors, not some generic thing for any kind of thing
happening.
2021-07-07 00:38:19 +03:00
Roman Khimov
5a9efcc654 stackitem: rework error handling
Return good named errors everywhere, export appropriate constants, make
errors.Is() work.
2021-07-07 00:18:00 +03:00
Roman Khimov
0de949b575 stackitem: remove Item/StackItem from function names
They're useless in a package named 'stackitem', make this package a bit more
user-friendly.
2021-07-06 19:56:23 +03:00
Roman Khimov
b9ff07f32c stackitem: add limit to serialized items
Standard binary serialization/deserialization is mostly used in VM to put/get
elements into/from storage, so they never should exceed MaxSize (otherwise one
won't be able to deserialize these items).

This patch leaves EncodeBinaryStackItem unprotected, but that's a streaming
interface, so it's up to the user of it to ensure its appropriate use (and our
uses are mostly for native contract's data, so they're fine).
2021-07-06 19:34:02 +03:00
Roman Khimov
8472064bbc stackitem: refactor serialization code, add explicit context 2021-07-06 19:32:52 +03:00
Roman Khimov
1b7b7e4bec stackitem: don't overwrite error when serializing Item
We must get out as quickly as possible.
2021-07-06 17:51:46 +03:00
Roman Khimov
e34fa2e915 stackitem: limit JSONization nesting level
We can have very deep reference types and attempt to JSONize them can easily
lead to OOM (even though there are no recursive references inside). Therefore
we have to limit them. While regular ToJSON() is buffer size limited to
MaxSize, ToJSONWithTypes is not and limiting it to MaxSize output will require
substantial rewriting effort while not really providing fair result, MaxSize
is more about stack item size while its JSON representation can be much bigger
because of various overheads.

Initial thought was to limit it by element count based on
MaxIteratorResultItems, but the problem here is that we don't always have this
limit in contexts using ToJSONWithTypes (like notification event
marshaling). Thus we need some generic limit which would be fine for all
users.

We at the same time have maxJSONDepth used when deserializing from JSON, so
it can be used for marshaling as well, it's not often that we have deeper
structures in real results.

Inspired by neo-project/neo#2521.
2021-07-06 17:33:16 +03:00
Roman Khimov
6879cbcdcc stackitem: properly detect recursive references during serialization
If we're done with element it no longer can lead to recursion error, so fix
cases like `[arr, arr]` where we have two copies of `arr` trigger this error
for no good reason (there is no recursion there).
2021-07-06 17:10:31 +03:00
Evgeniy Stratonikov
8d67a03aec stackitem: fix Buffer deserialization
Fix incorrect application log for transaction in N3 testnet
8eb4076f1f1c07e693eda7e810779488a2d2b50aba9b727fd237cbc3adbec9e9

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-06-29 11:39:55 +03:00
Anna Shaleva
f95bdd9cb5 vm: serialize + rune as \u002B for byte-contained stackitems
This commit fixes the following Go vs C# state diff:

block 74613: value mismatch for key EwAAAHN0cmVhbXMvDg==: eyJpZCI6MTQsInN0YXJ0IjoxNjIyNTAwMjAwMDAwLCJzdG9wIjoxNjIyNTAyMDAwMDAwLCJkZXBvc2l0IjoxMDAwMDAwMDAsInJlbWFpbmluZyI6MTAwMDAwMDAwLCJzZW5kZXIiOiJmeEY4RDl2ZFU3K0gwcDV3NTlyWllMNytNSlE9IiwicmVjaXBpZW50IjoiSVV6c3pveFV0S1NGVnlZRGczSmdTQTFlbTFNPSJ9 vs eyJpZCI6MTQsInN0YXJ0IjoxNjIyNTAwMjAwMDAwLCJzdG9wIjoxNjIyNTAyMDAwMDAwLCJkZXBvc2l0IjoxMDAwMDAwMDAsInJlbWFpbmluZyI6MTAwMDAwMDAwLCJzZW5kZXIiOiJmeEY4RDl2ZFU3XHUwMDJCSDBwNXc1OXJaWUw3XHUwMDJCTUpRPSIsInJlY2lwaWVudCI6IklVenN6b3hVdEtTRlZ5WURnM0pnU0ExZW0xTT0ifQ==

I.e.:
```
{"id":14,"start":1622500200000,"stop":1622502000000,"deposit":100000000,"remaining":100000000,"sender":"fxF8D9vdU7+H0p5w59rZYL7+MJQ=","recipient":"IUzszoxUtKSFVyYDg3JgSA1em1M="}
```
vs
```
{"id":14,"start":1622500200000,"stop":1622502000000,"deposit":100000000,"remaining":100000000,"sender":"fxF8D9vdU7\u002BH0p5w59rZYL7\u002BMJQ=","recipient":"IUzszoxUtKSFVyYDg3JgSA1em1M="}
```
2021-06-08 16:40:38 +03:00
Anna Shaleva
b14fc5da7c vm: increase waiting time for vm cli program 2021-06-01 16:29:08 +03:00
Anna Shaleva
e3611bfa4c vm, cli: allow to specify flags while loading VM 2021-05-28 12:07:41 +03:00
Roman Khimov
9d2712573f *: enable godot linter and fix all its warnings
It's important for NeoGo to have clean documentation. No functional changes.
2021-05-12 23:17:03 +03:00
Roman Khimov
c4e084b0d8 *: fix whitespace errors
leading/trailing newlines
2021-05-12 22:51:41 +03:00
Roman Khimov
de5e61588d vm: simplify some test code
gosimple: S1017: should replace this `if` statement with an unconditional `strings.TrimPrefix`
2021-05-12 18:44:35 +03:00
Roman Khimov
7af67d2a3c vm/cli: simplify buffer-related code
gosimple: S1030: should use buf.String() instead of string(buf.Bytes())
2021-05-12 18:42:50 +03:00
Roman Khimov
07cdbb815c *: drop unnecessary fmt.Sprintf uses
gosimple: S1039: unnecessary use of fmt.Sprintf
2021-05-12 18:29:39 +03:00
Anna Shaleva
6d59689d9c core: rename Neo.Crypto.CheckMultisig interop 2021-05-11 18:38:14 +03:00
Anna Shaleva
366e79b9b8 core: rename Neo.Crypto.CheckSig interop 2021-05-11 18:37:55 +03:00
Evgeniy Stratonikov
23a4e25436 interop: remove System.Iterator.Create, fix #1935
There are now only storage iterators. Related #1933.
2021-05-11 12:13:30 +03:00
Anna Shaleva
9eeebf481c rpc: allow to marshal Iterators for invoke* results 2021-04-30 16:23:06 +03:00
Evgeniy Stratonikov
8fa4a576f4 vm: allow Null arguments for LT,LE,GT,GE opcodes 2021-04-29 16:10:51 +03:00
Evgeniy Stratonikov
dc393642a2 opcode: fix GTE, LTE string representations 2021-04-29 16:08:48 +03:00
Anna Shaleva
f7dcb7ae29 vm: allow emit.Array handle uint256 2021-04-16 16:20:30 +03:00
Anna Shaleva
f57187e611 vm: throw unhandled exception with message instead of panic
The result is the same HALT state, but the exception message is the real
one got from user. Changes ported from C#:

1. Throw exception: 59b8ac73d2/src/neo-vm/ExecutionEngine.cs (L1448)
2. Prettify message: https://github.com/neo-project/neo-vm/blob/master/src/neo-vm/VMUnhandledException.cs#L28

The result is that instead of
```
2021-03-31T17:02:54.508+0300	WARN	contract invocation failed	{"tx": "2aefeb705f3a609df8767d9b45e036b9dd1eb77407e5732375981915668889b8", "block": 30640, "error": "error encountered at instruction 970 (THROW): runtime error: invalid memory address or nil pointer dereference"}
```

we'll get
```
2021-03-31T17:33:56.299+0300	WARN	contract invocation failed	{"tx": "2aefeb705f3a609df8767d9b45e036b9dd1eb77407e5732375981915668889b8", "block": 30640, "error": "error encountered at instruction 970 (THROW): unhandled exception: No authorization."}
```
in the node logs.
2021-03-31 19:37:52 +03:00
Anna Shaleva
793f27084b vm: specify syscall ID when panicing on syscall invocation
It's convinient to know the failing syscall without dumping
smartcontract instructions.
2021-03-31 19:00:59 +03:00
Anna Shaleva
2bb3ff2aff vm: return more detailed error from emit.Array 2021-03-26 20:44:32 +03:00
Evgeniy Stratonikov
b780a64b4d emit: allow to emit big.Int 2021-03-11 10:12:30 +03:00
Anna Shaleva
9015e50847 core: refactor Neo.Crypto.CheckMultisigWithECDsaSecpr1
Rename it to Neo.Crypto.CheckMultisig and remove `message` parameter.
2021-03-10 21:46:05 +03:00
Anna Shaleva
cdaca7be3e core: use Neo.Crypto.CheckSig for standard signature verification 2021-03-10 21:45:58 +03:00
Anna Shaleva
14ade42101 core: remove System.Binary.[Serialize, Deserialize] syscalls
And move their tests to native StdLib.
2021-03-10 19:24:19 +03:00
Evgeniy Stratonikov
ded6a70335 vm: add ParseSignatureContract() 2021-03-09 15:43:57 +03:00
Evgeniy Stratonikov
55009153a9 vm/emit: emit Boolean values correctly
We should convert both `true` and `false` values.
2021-03-09 13:34:22 +03:00
Roman Khimov
0c04b403b4
Merge pull request #1794 from nspcc-dev/fix/json
stackitem: escape control characters in `ToJSON()`
2021-03-02 13:37:27 +03:00
Evgeniy Stratonikov
e5fbf04529 stackitem: escape control characters in ToJSON()
Decoding uses `json.Decoder` which handles everything for us.
2021-03-02 10:47:02 +03:00
Roman Khimov
10cc19b33c
Merge pull request #1791 from nspcc-dev/drop-go-1-13
*: drop go 1.13
2021-03-01 17:20:26 +03:00
Anna Shaleva
2c81fc8b8e *: upgrade tests to use T.Cleanup() 2021-03-01 17:08:00 +03:00
Evgeniy Stratonikov
53327bf475 vm/testdata: update C# json tests 2021-03-01 16:58:53 +03:00
Evgeniy Stratonikov
d255c4a517 vm: implement SQRT opcode 2021-03-01 16:58:53 +03:00