Commit graph

3792 commits

Author SHA1 Message Date
Roman Khimov
11351b9702
Merge pull request #2114 from nspcc-dev/optimize-rpc
rpc/request: delay parameter unmarshaling
2021-08-13 16:30:42 +03:00
Evgeniy Stratonikov
3c34e6fa21 rpc/request: delay parameter unmarshaling
It is rather costly to try to unmarshal many structs in order.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-13 16:22:54 +03:00
Roman Khimov
5b12dd2025
Merge pull request #2128 from nspcc-dev/vm-update-int
Some VM optimizations
2021-08-13 16:16:01 +03:00
Evgeniy Stratonikov
6879f76a13 stackitem: make Buffer an alias to []byte
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-13 14:41:26 +03:00
Evgeniy Stratonikov
1dfef4ba26 stackitem: make ByteArray an alias to []byte
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-13 14:41:26 +03:00
Evgeniy Stratonikov
4f98ec2f53 vm: embed reference counter in compound items
```
name              old time/op  new time/op  delta
RefCounter_Add-8  44.8ns ± 4%  11.7ns ± 3%  -73.94%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-13 14:41:26 +03:00
Roman Khimov
adc660c3e0
Merge pull request #2123 from nspcc-dev/store-better
Store better
2021-08-13 12:50:24 +03:00
Evgeniy Stratonikov
dc9287bf5c compiler: use parameter directly in writeJumps
`Next` doesn't longer copy parameter.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-13 11:59:04 +03:00
Evgeniy Stratonikov
f5d1277bfd vm: do not copy parameter
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-13 11:52:38 +03:00
Evgeniy Stratonikov
e2910a7cb4 vm/cli: add public key -> address conversion, fix #2121
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-13 10:43:49 +03:00
Evgeniy Stratonikov
bb137abb03 crypto/keys: enforce length in PublicKey.DecodeBytes()
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-13 10:38:09 +03:00
Evgeniy Stratonikov
a5516e8c96 stackitem: make BigInteger alias to big.Int
Remove one indirection step.
``
name       old time/op    new time/op    delta
MakeInt-8    79.7ns ± 8%    56.2ns ± 8%  -29.44%  (p=0.000 n=10+10)

name       old alloc/op   new alloc/op   delta
MakeInt-8     48.0B ± 0%     40.0B ± 0%  -16.67%  (p=0.000 n=10+10)

name       old allocs/op  new allocs/op  delta
MakeInt-8      3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-12 17:53:36 +03:00
Evgeniy Stratonikov
cff8b1c24e stackitem: use Bool item directly
It is always copied.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-12 17:53:36 +03:00
Roman Khimov
ae071d4542 storage: introduce PutChangeSet and use it for Persist
We're using batches in wrong way during persist, we already have all changes
accumulated in two maps and then we move them to batch and then this is
applied. For some DBs like BoltDB this batch is just another MemoryStore, so
we essentially just shuffle the changeset from one map to another, for others
like LevelDB batch is just a serialized set of KV pairs, it doesn't help much
on subsequent PutBatch, we just duplicate the changeset again.

So introduce PutChangeSet that allows to take two maps with sets and deletes
directly. It also allows to simplify MemCachedStore logic.

neo-bench for single node with 10 workers, LevelDB:

  Reference:

  RPS    30189.132 30556.448 30390.482 ≈ 30379    ±  0.61%
  TPS    29427.344 29418.687 29434.273 ≈ 29427    ±  0.03%
  CPU %     33.304    27.179    33.860 ≈    31.45 ± 11.79%
  Mem MB   800.677   798.389   715.042 ≈   771    ±  6.33%

  Patched:

  RPS    30264.326 30386.364 30166.231 ≈ 30272    ± 0.36% ⇅
  TPS    29444.673 29407.440 29452.478 ≈ 29435    ± 0.08% ⇅
  CPU %     34.012    32.597    33.467 ≈   33.36  ± 2.14% ⇅
  Mem MB   549.126   523.656   517.684 ≈  530     ± 3.15% ↓ 31.26%

BoltDB:

  Reference:

  RPS    31937.647 31551.684 31850.408 ≈ 31780    ±  0.64%
  TPS    31292.049 30368.368 31307.724 ≈ 30989    ±  1.74%
  CPU %     33.792    22.339    35.887 ≈    30.67 ± 23.78%
  Mem MB  1271.687  1254.472  1215.639 ≈  1247    ±  2.30%

  Patched:

  RPS    31746.818 30859.485 31689.761 ≈ 31432    ± 1.58% ⇅
  TPS    31271.499 30340.726 30342.568 ≈ 30652    ± 1.75% ⇅
  CPU %     34.611    34.414    31.553 ≈    33.53 ± 5.11% ⇅
  Mem MB  1262.960  1231.389  1335.569 ≈  1277    ± 4.18% ⇅
2021-08-12 17:42:16 +03:00
Roman Khimov
5aff82aef4
Merge pull request #2119 from nspcc-dev/states-exchange/insole
core, network: prepare basis for Insole module
2021-08-12 10:35:02 +03:00
Roman Khimov
47f0f4c45f dao: completely drop Cached
It was very useful in 2.0 days, but today it only serves one purpose that
could easily (and more effectively!) be solved in another way.
2021-08-11 23:06:17 +03:00
Roman Khimov
3e60771175 core: deduplicate and simplify processNEP17Transfer a bit
Just refactoring, no functional changes.
2021-08-11 22:36:26 +03:00
Roman Khimov
50ee1a1f91 *: don't use dao.Cached in tests
There is no need to use it.
2021-08-11 21:02:50 +03:00
Roman Khimov
18682f2409 storage: don't use locks for memory batches
They're inherently single-threaded, so locking makes no sense for them.
2021-08-11 18:55:07 +03:00
Roman Khimov
13da1b62fb interop: fetch baseExecFee once and keep it in the Context
It never changes during single execution, so we can cache it and avoid going
to Policer via Chain for every instruction.
2021-08-11 15:42:23 +03:00
Roman Khimov
bdb2d24a5a vm: remove istack redirection in VM
VM always has istack and it doesn't even change, so doing this microallocation
makes no sense. Notice that estack is a bit harder to change we do replace it
in some cases and we compare pointers to it as well.
2021-08-11 14:42:01 +03:00
Roman Khimov
ff7d594bef vm: store refcounter directly in VM
VM always has it, so allocating yet another object makes no sense.
2021-08-11 13:25:58 +03:00
Anna Shaleva
0e3b9c48a2 core: add API to store StateSyncPoint and StateSyncCurrentBlockHeight
We need it in order not to mess up the blockchain which has its own
CurrentBlockHeight.
2021-08-10 14:06:28 +03:00
Anna Shaleva
cb01f533c0 core: store conflicting transactions in a separate method
(DAO).StoreConflictingTransactions will be reused from the state sync
module.
2021-08-10 13:47:13 +03:00
Anna Shaleva
72e654332e core: refactor block queue
It requires only two methods from Blockchainer: AddBlock and
BlockHeight. New interface will allow to easily reuse the block queue
for state exchange purposes.
2021-08-10 13:47:13 +03:00
Anna Shaleva
35501a281a core: remove untraceable blocks wrt StateSyncInterval 2021-08-10 13:47:10 +03:00
Roman Khimov
0a2bbf3c04
Merge pull request #2118 from nspcc-dev/neopt2
Networking improvements
2021-08-10 13:29:40 +03:00
Anna Shaleva
6ca7983be8 network: fix typo in error message 2021-08-10 11:00:39 +03:00
Anna Shaleva
76c687aaa1 config: add P2PStateExchangeExtensions and StateSyncInterval settings 2021-08-10 11:00:32 +03:00
Roman Khimov
1e0c70ecb0
Merge pull request #2117 from nspcc-dev/io-grow
Some io package improvements
2021-08-10 09:57:31 +03:00
Evgeniy Stratonikov
73e4040628 mpt: use BinWriter.Grow() instead of custom buffer
Also add benchmarks.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-10 09:34:05 +03:00
Evgeniy Stratonikov
c74de9a579 network: preallocate buffer for message
```
name            old time/op    new time/op    delta
MessageBytes-8     740ns ± 0%     684ns ± 2%   -7.58%  (p=0.000 n=10+10)

name            old alloc/op   new alloc/op   delta
MessageBytes-8    1.39kB ± 0%    1.20kB ± 0%  -13.79%  (p=0.000 n=10+10)

name            old allocs/op  new allocs/op  delta
MessageBytes-8      11.0 ± 0%      10.0 ± 0%   -9.09%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-10 09:33:52 +03:00
Evgeniy Stratonikov
dacf025dd9 io: add Grow to BinWriter
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 12:05:31 +03:00
Evgeniy Stratonikov
c69670c85b io: use a single slice for numbers
Slice takes 24 bytes of memory, while we really need only 9.
```
name                 old time/op    new time/op    delta
Transaction_Bytes-8     667ns ±17%     583ns ± 6%  -12.50%  (p=0.000 n=10+10)
GetVarSize-8            283ns ±11%     189ns ± 5%  -33.37%  (p=0.000 n=10+10)

name                 old alloc/op   new alloc/op   delta
Transaction_Bytes-8    1.01kB ± 0%    0.88kB ± 0%  -12.70%  (p=0.000 n=10+10)
GetVarSize-8             184B ± 0%       56B ± 0%  -69.57%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
Transaction_Bytes-8      7.00 ± 0%      6.00 ± 0%  -14.29%  (p=0.000 n=10+10)
GetVarSize-8             3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 12:04:28 +03:00
Evgeniy Stratonikov
620295efe3 transaction: add benchmark for transaction serialization
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 12:01:38 +03:00
Evgeniy Stratonikov
23adb1e2fc state: optimize NEP17TransferLog.Append
Do not allocate a separate buffer for the transfer.
```
name                       old time/op    new time/op    delta
NEP17TransferLog_Append-8    58.8µs ± 3%    32.1µs ± 1%  -45.40%  (p=0.000 n=10+9)

name                       old alloc/op   new alloc/op   delta
NEP17TransferLog_Append-8     118kB ± 1%      44kB ± 3%  -63.00%  (p=0.000 n=9+10)

name                       old allocs/op  new allocs/op  delta
NEP17TransferLog_Append-8       901 ± 1%       513 ± 3%  -43.08%  (p=0.000 n=9+8)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 11:09:49 +03:00
Evgeniy Stratonikov
403a4b75de state/test: add benchmark for NEP17TransferLog.Append
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 11:09:49 +03:00
Evgeniy Stratonikov
b210a34b1e state: optimize NEP17Balance deserialization
```
BenchmarkNEP17BalanceFromBytes/stackitem-8         	 2402318	       503.3 ns/op	     208 B/op	      10 allocs/op
BenchmarkNEP17BalanceFromBytes/from_bytes-8        	 7623139	       160.7 ns/op	      72 B/op	       3 allocs/op
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 11:09:49 +03:00
Evgeniy Stratonikov
3218b74ea5 state: optimize NEP17Balance serialization
Put to slice directly and allow to provide pre-allocated buffer.
```
BenchmarkNEP17BalanceBytes/stackitem-8         	 1712475	       673.4 ns/op	     448 B/op	       9 allocs/op
BenchmarkNEP17BalanceBytes/bytes-8             	13422715	        75.80 ns/op	      32 B/op	       2 allocs/op
BenchmarkNEP17BalanceBytes/bytes,_prealloc-8   	25990371	        46.46 ns/op	      16 B/op	       1 allocs/op
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-09 11:09:06 +03:00
Roman Khimov
7bb82f1f99 network: merge two loops in iteratePeersWithSendMsg, send to 2/3
Refactor code and be fine with sending to just 2/3 of proper peers. Previously
it was an edge case, but it can be a normal thing to do also as broadcasting
to everyone is obviously too expensive and excessive (hi, #608).

Baseline (four node, 10 workers):

RPS    8180.760 8137.822 7858.358 7820.011 8051.076 ≈ 8010   ± 2.04%
TPS    7819.831 7521.172 7519.023 7242.965 7426.000 ≈ 7506   ± 2.78%
CPU %    41.983   38.775   40.606   39.375   35.537 ≈   39.3 ± 6.15%
Mem MB 2947.189 2743.658 2896.688 2813.276 2863.108 ≈ 2853   ± 2.74%

Patched:

RPS    9714.567 9676.102 9358.609 9371.408 9301.372 ≈ 9484   ±  2.05% ↑ 18.40%
TPS    8809.796 8796.854 8534.754 8661.158 8426.162 ≈ 8646   ±  1.92% ↑ 15.19%
CPU %    44.980   45.018   33.640   29.645   43.830 ≈   39.4 ± 18.41% ↑  0.25%
Mem MB 2989.078 2976.577 2306.185 2351.929 2910.479 ≈ 2707   ± 12.80% ↓  5.12%

There is a nuance with this patch however. While typically it works the way
outlined above, sometimes it works like this:

RPS ≈ 6734.368
TPS ≈ 6299.332
CPU ≈ 25.552%
Mem ≈ 2706.046MB

And that's because the log looks like this:

DeltaTime, TransactionsCount, TPS
5014, 44212, 8817.710
5163, 49690, 9624.249
5166, 49523, 9586.334
5189, 49693, 9576.604
5198, 49339, 9491.920
5147, 49559, 9628.716
5192, 49680, 9568.567
5163, 49750, 9635.871
5183, 49189, 9490.450
5159, 49653, 9624.540
5167, 47945, 9279.079
5179, 2051, 396.022
5015, 4, 0.798
5004, 0, 0.000
5003, 0, 0.000
5003, 0, 0.000
5003, 0, 0.000
5003, 0, 0.000
5004, 0, 0.000
5003, 2925, 584.649
5040, 49099, 9741.865
5161, 49718, 9633.404
5170, 49228, 9521.857
5179, 49773, 9610.543
5167, 47253, 9145.152
5202, 49788, 9570.934
5177, 47704, 9214.603
5209, 46610, 8947.975
5249, 49156, 9364.831
5163, 18284, 3541.352
5072, 174, 34.306

On a network with 4 CNs and 1 RPC node there is 1/256 probability that a block
won't be broadcasted to RPC node, so it won't see it until ping timeout kicks
in. While it doesn't see a block it can't accept new incoming transactions so
the bench gets stuck basically. To me that's an acceptable trade-off because
normal networks are much larger than that and the effect of this patch is way
more important there, but still that's what we have and we need to take into
account.
2021-08-06 21:10:34 +03:00
Roman Khimov
966a16e80e network: keep track of dead peers in iteratePeersWithSendMsg()
send() can return errStateMismatch, errGone and errBusy. errGone means the
peer is dead and it won't ever be active again, it doesn't make sense retrying
sends to it. errStateMismatch is technically "not yet ready", but we can't
wait for it either, no one knows how much will it take to complete
handshake. So only errBusy means we can retry.

So keep track of dead peers and adjust tries counting appropriately.
2021-08-06 21:10:34 +03:00
Roman Khimov
80f3ec2312 network: move peer filtering to getPeers()
It doesn't change much, we can't magically get more valid peers and if some
die while we're iterating we'd detect that by an error returned from send().
2021-08-06 21:10:34 +03:00
Roman Khimov
de6f4987f6 network: microoptimize iteratePeersWithSendMsg()
Now that s.getPeers() returns a slice we can use slice for `success` too, maps
are more expensive.
2021-08-06 21:10:34 +03:00
Roman Khimov
d51db20405 network: randomize peer iteration order
While iterating over map in getPeers() is non-deterministic it's not really
random enough for our purposes (usually maps have 2-3 paths through them), we
need to fill our peers queues more uniformly.

Believe it or not, but it does affect performance metrics, baseline (four
nodes, 10 workers):

RPS ≈  7791.675 7996.559 7834.504 7746.705 7891.614 ≈ 7852   ±  1.10%
TPS ≈  7241.497 7711.765 7520.211 7425.890 7334.443 ≈ 7447   ±  2.17%
CPU %    29.853   39.936   39.945   36.371   39.999 ≈   37.2 ± 10.57%
Mem MB 2749.635 2791.609 2828.610 2910.431 2863.344 ≈ 2829   ±  1.97%

Patched:

RPS    8180.760 8137.822 7858.358 7820.011 8051.076 ≈ 8010   ± 2.04% ↑ 2.01%
TPS    7819.831 7521.172 7519.023 7242.965 7426.000 ≈ 7506   ± 2.78% ↑ 0.79%
CPU %    41.983   38.775   40.606   39.375   35.537 ≈   39.3 ± 6.15% ↑ 5.65%
Mem MB 2947.189 2743.658 2896.688 2813.276 2863.108 ≈ 2853   ± 2.74% ↑ 0.85%
2021-08-06 21:10:34 +03:00
Roman Khimov
b55c75d59d network: hide Peers, make it return a slice
Slice is a bit more efficient, we don't need a map for Peers() users and it's
not really interesting to outside users, so better hide this method.
2021-08-06 21:10:34 +03:00
Roman Khimov
119b4200ac network: add fail-fast route for tx double processing
When transaction spreads through the network many nodes are likely to get it
in roughly the same time. They will rebroadcast it also in roughly the same
time. As we have a number of peers it's quite likely that we'd get an Inv with
the same transaction from multiple peers simultaneously. We will ask them for
this transaction (independently!) and again we're likely to get it in roughly
the same time. So we can easily end up with multiple threads processing the
same transaction. Only one will succeed, but we can actually easily avoid
doing it in the first place saving some CPU cycles for other things.

Notice that we can't do it _before_ receiving a transaction because nothing
guarantees that the peer will respond to our transaction request, so
communication overhead is unavoidable at the moment, but saving on processing
already gives quite interesting results.

Baseline, four nodes with 10 workers:

RPS    7176.784 7014.511 6139.663 7191.280 7080.852 ≈ 6921   ± 5.72%
TPS    6945.409 6562.756 5927.050 6681.187 6821.794 ≈ 6588   ± 5.38%
CPU %    44.400   43.842   40.418   49.211   49.370 ≈   45.4 ± 7.53%
Mem MB 2693.414 2640.602 2472.007 2731.482 2707.879 ≈ 2649   ± 3.53%

Patched:

RPS ≈  7791.675 7996.559 7834.504 7746.705 7891.614 ≈ 7852   ±  1.10% ↑ 13.45%
TPS ≈  7241.497 7711.765 7520.211 7425.890 7334.443 ≈ 7447   ±  2.17% ↑ 13.04%
CPU %    29.853   39.936   39.945   36.371   39.999 ≈   37.2 ± 10.57% ↓ 18.06%
Mem MB 2749.635 2791.609 2828.610 2910.431 2863.344 ≈ 2829   ±  1.97% ↑  6.80%
2021-08-06 21:10:25 +03:00
Roman Khimov
7fc153ed2a network: only ask mempool for intersections with received Inv
Most of the time on healthy network we see new transactions appearing that are
not present in the mempool. Once they get into mempool we don't ask for them
again when some other peer sends an Inv with them. Then these transactions are
usually added into block, removed from mempool and no one actually sends them
again to us. Some stale nodes can do that, but it's not very likely to
happen.

At the receiving end at the same time it's quite expensive to do full chain
HasTransaction() query, so if we can avoid doing that it's always good. Here
it technically allows resending old transaction that will be re-requested and
an attempt to add it to mempool will be made. But it'll inevitably fail
because the same HasTransaction() check is done there too. One can try to
maliciously flood the node with stale transactions but it doesn't differ from
flooding it with any other invalid transactions, so there is no new attack
vector added.

Baseline, 4 nodes with 10 workers:

RPS    6902.296 6465.662 6856.044 6785.515 6157.024 ≈ 6633   ± 4.26%
TPS    6468.431 6218.867 6610.565 6288.596 5790.556 ≈ 6275   ± 4.44%
CPU %    50.231   42.925   49.481   48.396   42.662 ≈   46.7 ± 7.01%
Mem MB 2856.841 2684.103 2756.195 2733.485 2422.787 ≈ 2691   ± 5.40%

Patched:

RPS    7176.784 7014.511 6139.663 7191.280 7080.852 ≈ 6921   ± 5.72% ↑ 4.34%
TPS    6945.409 6562.756 5927.050 6681.187 6821.794 ≈ 6588   ± 5.38% ↑ 4.99%
CPU %    44.400   43.842   40.418   49.211   49.370 ≈   45.4 ± 7.53% ↓ 2.78%
Mem MB 2693.414 2640.602 2472.007 2731.482 2707.879 ≈ 2649   ± 3.53% ↓ 1.56%
2021-08-06 20:53:02 +03:00
Roman Khimov
f78bd6474f network: handle incoming message in a separate goroutine
Network communication takes time. Handling some messages (like transaction)
also takes time. We can share this time by making handler a separate
goroutine. So while message is being handled receiver can already get and
parse the next one.

It doesn't improve metrics a lot, but still I think it makes sense and in some
scenarios this can be more beneficial than this.

e41fc2fd1b, 4 nodes, 10 workers

RPS    6732.979 6396.160 6759.624 6246.398 6589.841 ≈ 6545   ± 3.02%
TPS    6491.062 5984.190 6275.652 5867.477 6360.797 ≈ 6196   ± 3.77%
CPU %    42.053   43.515   44.768   40.344   44.112 ≈   43.0 ± 3.69%
Mem MB 2564.130 2744.236 2636.267 2589.505 2765.926 ≈ 2660   ± 3.06%

Patched:

RPS    6902.296 6465.662 6856.044 6785.515 6157.024 ≈ 6633   ± 4.26% ↑ 1.34%
TPS    6468.431 6218.867 6610.565 6288.596 5790.556 ≈ 6275   ± 4.44% ↑ 1.28%
CPU %    50.231   42.925   49.481   48.396   42.662 ≈   46.7 ± 7.01% ↑ 8.60%
Mem MB 2856.841 2684.103 2756.195 2733.485 2422.787 ≈ 2691   ± 5.40% ↑ 1.17%
2021-08-06 19:37:37 +03:00
Roman Khimov
b989504d74
Merge pull request #2108 from nspcc-dev/optimize-mpt
Some allocation optimizations
2021-08-06 14:51:10 +03:00
Evgeniy Stratonikov
bd2b1a0521 mpt: add Size method to trie nodes
Knowing serialized size of the node is useful for
preallocating byte-slice in advance.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-06 12:01:16 +03:00
Evgeniy Stratonikov
db80ef28df mpt: move empty hash node in a separate type
We use them quite frequently (consider children for a new branch
node) and it is better to get rid of unneeded allocations.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-06 12:01:16 +03:00
Evgeniy Stratonikov
f02d8b4ec4 stackitem: serialize integers to the pre-allocated slice
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-06 11:59:24 +03:00
Evgeniy Stratonikov
291a29af1e *: do not use WriteArray for frequently used items
`WriteArray` involves reflection, it makes sense to optimize
serialization of transactions and application logs which are serialized
constantly. Adding case in a type switch in `WriteArray` is not an
option because we don't want new dependencies for `io` package.

```
name                          old time/op    new time/op    delta
AppExecResult_EncodeBinary-8     852ns ± 3%     656ns ± 2%  -22.94%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
AppExecResult_EncodeBinary-8      448B ± 0%      376B ± 0%  -16.07%  (p=0.000 n=10+10)

name                          old allocs/op  new allocs/op  delta
AppExecResult_EncodeBinary-8      7.00 ± 0%      5.00 ± 0%  -28.57%  (p=0.000 n=10+10)
```

```
name                 old time/op    new time/op    delta
Transaction_Bytes-8    1.29µs ± 3%    0.76µs ± 5%  -41.52%  (p=0.000 n=9+10)

name                 old alloc/op   new alloc/op   delta
Transaction_Bytes-8    1.21kB ± 0%    1.01kB ± 0%  -16.56%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
Transaction_Bytes-8      12.0 ± 0%       7.0 ± 0%  -41.67%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-06 11:59:20 +03:00
Roman Khimov
95e1f5f77b
Merge pull request #2113 from nspcc-dev/optimize-witness-hashing
core: don't recalculate witness script hash
2021-08-06 11:57:54 +03:00
Roman Khimov
f9663a97a1 network: fix Ping messages
* NewPing() accepts block index first and nonce then.
 * Block height should be used, it'll be important for state exchanging nodes
2021-08-06 11:28:09 +03:00
Roman Khimov
39f874d03f core: don't recalculate witness script hash
We know it already, but with current loading code VM will hash it once
more. It doesn't help a lot and still it costs nothing to avoid this
overhead.

name             old time/op    new time/op    delta
VerifyWitness-8    93.4µs ± 3%    92.7µs ± 2%    ~     (p=0.353 n=10+10)

name             old alloc/op   new alloc/op   delta
VerifyWitness-8    3.43kB ± 0%    3.40kB ± 0%  -0.70%  (p=0.000 n=9+9)

name             old allocs/op  new allocs/op  delta
VerifyWitness-8      67.0 ± 0%      66.0 ± 0%  -1.49%  (p=0.000 n=10+10)
2021-08-06 11:25:09 +03:00
Evgeniy Stratonikov
43ee671f36 mpt: do not allocate NodeObject for serialization
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-06 10:28:19 +03:00
Roman Khimov
e41fc2fd1b
Merge pull request #2111 from nspcc-dev/drop-refuel
native: drop Refuel method from GAS
2021-08-05 16:42:29 +03:00
Roman Khimov
d6bd6b6888 native: drop Refuel method from GAS
It can be used to attack the network (amplifying DOS), so it's broken
beyond repair. This reverts ac601601c1.

See also neo-project/neo#2560 and neo-project/neo#2561.
2021-08-05 10:27:13 +03:00
Roman Khimov
1b186e046b network: use optimized decoder for transactions
NewTransactionFromBytes() works a bit faster and uses less memory.
2021-08-04 23:49:07 +03:00
Roman Khimov
892c9785ad transaction: don't allocate new buffer to calculate hash
We can write directly to hash.Hash.

name               old time/op    new time/op    delta
DecodeBinary-8       2.89µs ± 3%    2.82µs ± 5%     ~     (p=0.052 n=10+10)
DecodeJSON-8         13.0µs ± 1%    12.8µs ± 1%   -1.54%  (p=0.002 n=10+8)
DecodeFromBytes-8    2.37µs ± 1%    2.25µs ± 5%   -5.25%  (p=0.000 n=9+10)

name               old alloc/op   new alloc/op   delta
DecodeBinary-8       1.75kB ± 0%    1.53kB ± 0%  -12.79%  (p=0.000 n=10+10)
DecodeJSON-8         3.49kB ± 0%    3.26kB ± 0%   -6.42%  (p=0.000 n=10+10)
DecodeFromBytes-8    1.37kB ± 0%    1.14kB ± 0%  -16.37%  (p=0.000 n=10+10)

name               old allocs/op  new allocs/op  delta
DecodeBinary-8         26.0 ± 0%      23.0 ± 0%  -11.54%  (p=0.000 n=10+10)
DecodeJSON-8           58.0 ± 0%      55.0 ± 0%   -5.17%  (p=0.000 n=10+10)
DecodeFromBytes-8      18.0 ± 0%      15.0 ± 0%  -16.67%  (p=0.000 n=10+10)
2021-08-04 23:43:20 +03:00
Roman Khimov
6d10cdc2f6 transaction: avoid ReadArray()
Reflection adds some real cost to it:

name               old time/op    new time/op    delta
DecodeBinary-8       3.14µs ± 5%    2.89µs ± 3%   -8.19%  (p=0.000 n=10+10)
DecodeJSON-8         12.6µs ± 3%    13.0µs ± 1%   +3.77%  (p=0.000 n=10+10)
DecodeFromBytes-8    2.73µs ± 2%    2.37µs ± 1%  -13.12%  (p=0.000 n=9+9)

name               old alloc/op   new alloc/op   delta
DecodeBinary-8       1.82kB ± 0%    1.75kB ± 0%   -3.95%  (p=0.000 n=10+10)
DecodeJSON-8         3.49kB ± 0%    3.49kB ± 0%     ~     (all equal)
DecodeFromBytes-8    1.44kB ± 0%    1.37kB ± 0%   -5.00%  (p=0.000 n=10+10)

name               old allocs/op  new allocs/op  delta
DecodeBinary-8         29.0 ± 0%      26.0 ± 0%  -10.34%  (p=0.000 n=10+10)
DecodeJSON-8           58.0 ± 0%      58.0 ± 0%     ~     (all equal)
DecodeFromBytes-8      21.0 ± 0%      18.0 ± 0%  -14.29%  (p=0.000 n=10+10)
2021-08-04 23:34:57 +03:00
Roman Khimov
d2732a71d8 transaction: don't overwrite error and witnesses length check
ReadArray() can return some error and we shouldn't overwrite it. At the same
time limiting ReadArray() to the number of Signers can make it return wrong
error if the number of witnesses actually is bigger than the number of
signers, so use MaxAttributes.
2021-08-04 23:17:50 +03:00
Roman Khimov
d487b54612 transaction: don't recalculate size when decoding from buffer
name               old time/op    new time/op    delta
DecodeBinary-8       3.17µs ± 6%    3.14µs ± 5%     ~     (p=0.579 n=10+10)
DecodeJSON-8         12.8µs ± 3%    12.6µs ± 3%     ~     (p=0.105 n=10+10)
DecodeFromBytes-8    3.45µs ± 4%    2.73µs ± 2%  -20.70%  (p=0.000 n=10+9)

name               old alloc/op   new alloc/op   delta
DecodeBinary-8       1.82kB ± 0%    1.82kB ± 0%     ~     (all equal)
DecodeJSON-8         3.49kB ± 0%    3.49kB ± 0%     ~     (all equal)
DecodeFromBytes-8    1.82kB ± 0%    1.44kB ± 0%  -21.05%  (p=0.000 n=10+10)

name               old allocs/op  new allocs/op  delta
DecodeBinary-8         29.0 ± 0%      29.0 ± 0%     ~     (all equal)
DecodeJSON-8           58.0 ± 0%      58.0 ± 0%     ~     (all equal)
DecodeFromBytes-8      29.0 ± 0%      21.0 ± 0%  -27.59%  (p=0.000 n=10+10)
2021-08-04 23:13:58 +03:00
Roman Khimov
64c780ad7a native: optimize totalSupply operations during token burn/mint
We burn GAS in OnPersist for every transaction so some buffer reuse here is
quite natural.

This also doesn't change a lot in the overall TPS picture, maybe adding some
1%.
2021-08-03 17:59:38 +03:00
Roman Khimov
dede4fa7b1 state: convert NEO balance to stack item directly
Avoid calling Append() that will reallocate the slice, we know the length of
the slice exactly.
2021-08-03 17:59:38 +03:00
Roman Khimov
5c65d33439 native: move required balance check to token contracts
Which duplicates the check, but deduplicates error path. This check forced
double balance deserialization which is quite costly operation, so we better
do it once.

It's hardly noticeable as of TPS metrics though, maybe some 1-2%%.
2021-08-03 17:59:38 +03:00
Roman Khimov
85936de254 vm: don't create reference counter when it's not needed
* invocation stack doesn't need refcounting
 * exception stack doesn't need refcounting
 * evaluation stack always has VM-level refcounter
2021-08-02 22:38:41 +03:00
Roman Khimov
2c2ccdca74 opcode: optimize IsValid
Map access costs much more than array access.

name       old time/op  new time/op  delta
IsValid-8  17.6ns ± 2%   1.1ns ± 2%  -93.68%  (p=0.008 n=5+5)
2021-08-02 21:46:19 +03:00
Roman Khimov
3c1325035e fee: use array for opcodes
Use less memory and have faster access.

name       old time/op  new time/op  delta
Opcode1-8  22.4ns ± 6%   3.0ns ± 6%  -86.63%  (p=0.000 n=10+10)
2021-08-02 20:18:33 +03:00
Roman Khimov
dfc514eda0
Merge pull request #2102 from nspcc-dev/store4
Improve (*MemCachedStore).Persist
2021-08-02 20:10:05 +03:00
Roman Khimov
82f481e143
Merge pull request #2105 from nspcc-dev/json-restrict
native/std: restrint amount of items in JSON deserialization
2021-08-02 19:41:54 +03:00
Evgeniy Stratonikov
bdb9748c1b native/std: restrict amount of items in JSON deserialization
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-08-02 18:57:47 +03:00
Roman Khimov
f8174ca64c core: ensure data logged is from persistent store
Using bc.dao here is wrong, it can contain unpersisted data.
2021-08-02 16:33:09 +03:00
Roman Khimov
8277b7a19a core: don't spawn goroutine for persist function
It doesn't make any sense, in some situations it leads to a number of
goroutines created that will Persist one after another (as we can't Persist
concurrently). We can manage it better in a single thread.

This doesn't change performance in any way, but somewhat reduces resource
consumption. It was tested neo-bench (single node, 10 workers, LevelDB) on two
machines and block dump processing (RC4 testnet up to 62800 with VerifyBlocks
set to false) on i7-8565U.

Reference (b9be892bf9):

Ryzen 9 5950X:
RPS     27747.349 27407.726 27520.210  ≈ 27558   ± 0.63%
TPS     26992.010 26993.468 27010.966  ≈ 26999   ± 0.04%
CPU %      28.928    28.096    29.105  ≈    28.7 ± 1.88%
Mem MB    760.385   726.320   756.118  ≈   748   ± 2.48%

Core i7-8565U:
RPS     7783.229 7628.409 7542.340  ≈ 7651   ± 1.60%
TPS     7708.436 7607.397 7489.459  ≈ 7602   ± 1.44%
CPU %     74.899   71.020   72.697  ≈   72.9 ± 2.67%
Mem MB   438.047  436.967  416.350  ≈  430   ± 2.84%

DB restore:
real    0m20.838s 0m21.895s 0m21.794s  ≈ 21.51 ± 2.71%
user    0m39.091s 0m40.565s 0m41.493s  ≈ 40.38 ± 3.00%
sys      0m3.184s  0m2.923s  0m3.062s  ≈  3.06 ± 4.27%

Patched:

Ryzen 9 5950X:
RPS     27636.957 27246.911 27462.036  ≈ 27449   ±  0.71%  ↓ 0.40%
TPS     27003.672 26993.468 27011.696  ≈ 27003   ±  0.03%  ↑ 0.01%
CPU %      28.562    28.475    28.012  ≈    28.3 ±  1.04%  ↓ 1.39%
Mem MB    627.007   648.110   794.895  ≈   690   ± 13.25%  ↓ 7.75%

Core i7-8565U:
RPS     7497.210 7527.797 7897.532  ≈ 7641   ±  2.92%  ↓ 0.13%
TPS     7461.128 7482.678 7841.723  ≈ 7595   ±  2.81%  ↓ 0.09%
CPU %     71.559   73.423   69.005  ≈   71.3 ±  3.11%  ↓ 2.19%
Mem MB   393.090  395.899  482.264  ≈  424   ± 11.96%  ↓ 1.40%

DB restore:
real    0m20.773s 0m21.583s 0m20.522s  ≈ 20.96 ±  2.65%  ↓ 2.56%
user    0m39.322s 0m42.268s 0m38.626s  ≈ 40.07 ±  4.82%  ↓ 0.77%
sys      0m3.006s  0m3.597s  0m3.042s  ≈  3.22 ± 10.31%  ↑ 5.23%
2021-08-02 16:33:00 +03:00
Roman Khimov
b9be892bf9 storage: allow accessing MemCachedStore during Persist
Persist by its definition doesn't change MemCachedStore visible state, all KV
pairs that were acessible via it before Persist remain accessible after
Persist. The only thing it does is flushing of the current set of KV pairs
from memory to peristent store. To do that it needs read-only access to the
current KV pair set, but technically it then replaces maps, so we have to use
full write lock which makes MemCachedStore inaccessible for the duration of
Persist. And Persist can take a lot of time, it's about disk access for
regular DBs.

What we do here is we create new in-memory maps for MemCachedStore before
flushing old ones to the persistent store. Then a fake persistent store is
created which actually is a MemCachedStore with old maps, so it has exactly
the same visible state. This Store is never accessed for writes, so we can
read it without taking any internal locks and at the same time we no longer
need write locks for original MemCachedStore, we're not using it. All of this
makes it possible to use MemCachedStore as normally reads are handled going
down to whatever level is needed and writes are handled by new maps. So while
Persist for (*Blockchain).dao does its most time-consuming work we can process
other blocks (reading data for transactions and persisting storeBlock caches
to (*Blockchain).dao).

The change was tested for performance with neo-bench (single node, 10 workers,
LevelDB) on two machines and block dump processing (RC4 testnet up to 62800
with VerifyBlocks set to false) on i7-8565U.

Reference results (bbe4e9cd7b):

Ryzen 9 5950X:
RPS     23616.969 22817.086 23222.378  ≈ 23218   ± 1.72%
TPS     23047.316 22608.578 22735.540  ≈ 22797   ± 0.99%
CPU %      23.434    25.553    23.848  ≈    24.3 ± 4.63%
Mem MB    600.636   503.060   582.043  ≈   562   ± 9.22%

Core i7-8565U:
RPS     6594.007 6499.501 6572.902  ≈ 6555   ± 0.76%
TPS     6561.680 6444.545 6510.120  ≈ 6505   ± 0.90%
CPU %     58.452   60.568   62.474    ≈ 60.5 ± 3.33%
Mem MB   234.893  285.067  269.081   ≈ 263   ± 9.75%

DB restore:
real    0m22.237s 0m23.471s 0m23.409s  ≈ 23.04 ± 3.02%
user    0m35.435s 0m38.943s 0m39.247s  ≈ 37.88 ± 5.59%
sys      0m3.085s  0m3.360s  0m3.144s  ≈  3.20 ± 4.53%

After the change:

Ryzen 9 5950X:
RPS     27747.349 27407.726 27520.210  ≈ 27558   ± 0.63%  ↑ 18.69%
TPS     26992.010 26993.468 27010.966  ≈ 26999   ± 0.04%  ↑ 18.43%
CPU %      28.928    28.096    29.105  ≈    28.7 ± 1.88%  ↑ 18.1%
Mem MB    760.385   726.320   756.118  ≈   748   ± 2.48%  ↑ 33.10%

Core i7-8565U:
RPS     7783.229 7628.409 7542.340  ≈ 7651   ± 1.60%  ↑ 16.72%
TPS     7708.436 7607.397 7489.459  ≈ 7602   ± 1.44%  ↑ 16.85%
CPU %     74.899   71.020   72.697  ≈   72.9 ± 2.67%  ↑ 20.50%
Mem MB   438.047  436.967  416.350  ≈  430   ± 2.84%  ↑ 63.50%

DB restore:
real    0m20.838s 0m21.895s 0m21.794s  ≈ 21.51 ± 2.71%  ↓ 6.64%
user    0m39.091s 0m40.565s 0m41.493s  ≈ 40.38 ± 3.00%  ↑ 6.60%
sys      0m3.184s  0m2.923s  0m3.062s  ≈  3.06 ± 4.27%  ↓ 4.38%

It obviously uses more memory now and utilizes CPU more aggressively, but at
the same time it allows to improve all relevant metrics and finally reach a
situation where we process 50K transactions in less than second on Ryzen 9
5950X (going higher than 25K TPS). The other observation is much more stable
block time, on Ryzen 9 it's as close to 1 second as it could be.
2021-08-02 16:33:00 +03:00
Roman Khimov
3cebd2b129 interop: use non-Cached wrapped DAO
Cached only caches NEP-17 tracking data now, it makes no sense here.
2021-07-30 15:45:17 +03:00
Roman Khimov
fa7314ea90 dao: drop dropNEP17Cache from Cached
It's not used now.
2021-07-30 15:45:17 +03:00
Roman Khimov
49be753850 core: spread storeBlock actions to three goroutines
Block processing consists of:
 * saving block/transactions to the DB
 * executing blocks/transactions
 * processing notifications/saving AERs
 * updating MPT
 * atomically updating Blockchain state

Of these the first one is completely independent of others, it can be done in
a separate routine easily. The third one technically depends on the second,
it just doesn't have data until something is executed. At the same time it
doesn't affect future executions in any way, so we can offload
AER/notification processing to separate goroutine (while the main thread
proceeds with other transactions).

MPT update depends on all executions, so it can't be offloaded, but it can be
done concurrently to AER processing. And only the last thing actually needs
all previous ones to be finished, so it's a natural synchronization point.

So we spawn two additional routines and let the main one execute transactions
and update MPT as fast as it can. While technically all of these routines
could share single DAO (they are working with different KV sets) benchmarking
shows that using separate DAOs and then persisting them to lower one actually
works about 7-8%% better. At the same time we can simplify DAOs used, Cached
one is only relevant for AER processing because it caches NEP-17 tracking
data, everything else can do just fine with Simple.

The change was tested for performance with neo-bench (single node, 10 workers,
LevelDB) on two machines and block dump processing (RC4 testnet up to 50825
with VerifyBlocks set to false) on i7-8565U. neo-bench creates huge blocks
with lots of transactions while RC4 dump mostly consists of empty blocks.

Reference results (06c3dda5d1):

Ryzen 9 5950X:
RPS ≈ 20059.569   21186.328   20158.983   ≈ 20468   ±  3.05%
TPS ≈ 19544.993   20585.450   19658.338   ≈ 19930   ±  2.86%
CPU ≈    18.682%     23.877%     22.852%  ≈    21.8 ± 12.62%
Mem ≈   618.981MB   559.246MB   541.539MB ≈   573   ±  7.08%

Core i7-8565U:
RPS ≈ 5927.082   6526.739   6372.115   ≈ 6275   ± 4.96%
TPS ≈ 5899.531   6477.187   6329.515   ≈ 6235   ± 4.81%
CPU ≈   56.346%    61.955%    58.125%  ≈   58.8 ± 4.87%
Mem ≈  212.191MB  224.974MB  205.479MB ≈  214   ± 4.62%

DB restore:
real    0m12.683s 0m13.222s 0m13.382s  ≈ 13.096 ±  2.80%
user    0m18.501s 0m19.163s 0m19.489s  ≈ 19.051 ±  2.64%
sys      0m1.404s  0m1.396s  0m1.666s  ≈  1.489 ± 10.32%

After the change:

Ryzen 9 5950X:
RPS ≈ 23056.899   22822.015   23006.543   ≈ 22962   ± 0.54%
TPS ≈ 22594.785   22292.071   22800.857   ≈ 22562   ± 1.13%
CPU ≈    24.262%     23.185%     25.921%  ≈    24.5 ± 5.65%
Mem ≈   614.254MB   613.204MB   555.491MB ≈   594   ± 5.66%

Core i7-8565U:
RPS ≈ 6378.702   6423.927   6363.788      ≈ 6389   ± 0.49%
TPS ≈ 6327.072   6372.552   6311.179      ≈ 6337   ± 0.50%
CPU ≈   57.599%    58.622%    59.737%     ≈   58.7 ± 1.82%
Mem ≈  198.697MB  188.746MB  200.235MB    ≈  196   ± 3.18%

DB restore:
real    0m13.576s 0m13.334s 0m12.757s  ≈  13.222 ±  3.18%
user    0m19.113s 0m19.490s 0m20.197s  ≈  19.600 ±  2.81%
sys      0m2.211s  0m1.558s  0m1.559s  ≈   1.776 ± 21.21%

On Ryzen 9 we've got 12% better RPS, 13% better TPS with 12% CPU and 3% RAM
more used. Core i7-8565U changes don't seem to be statistically significant:
1.8% more RPS, 1.6% more TPS with about the same CPU and 8.5% less RAM
used. It also is 1% worse in DB restore time.

The result is somewhat expected, on a powerful machine with lots of spare
cores we get 10%+ better results while on average resource-constrained laptop it
doesn't change much (the machine is already saturated). Overall, this seems to
be worthwhile.
2021-07-30 15:45:17 +03:00
Roman Khimov
06c3dda5d1
Merge pull request #2093 from nspcc-dev/states-exchange/drop-nep17-balance-state
core: implement dynamic NEP17 balances tracking
2021-07-29 19:08:42 +03:00
Evgeniy Stratonikov
283173bb9d wallet: use named constants in Seek
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-29 17:11:50 +03:00
Evgeniy Stratonikov
a429aa3e68 wallet: truncate file when writing
If wallet size decreases, we need to remove trailing garbage if it
exists. This can happen when removing account or reading pretty-printed
wallet. It doesn't affect our CLI (we decode only file prefix), but
it is nice to always have a valid JSON file.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-29 17:11:49 +03:00
Evgeniy Stratonikov
8f196c8222 wallet: marshal before writing to file
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-29 16:07:36 +03:00
Anna Shaleva
a30e48ff90 core: increment the DB version
DB scheme has been changed.
2021-07-29 10:23:13 +03:00
Anna Shaleva
e8bed184d5 core: implement dynamic NEP17 balances tracking
Request NEP17 balances from a set of NEP17 contracts instead of getting
them from storage. LastUpdatedBlock tracking remains untouched, because
there's no way to retrieve it dynamically.
2021-07-29 10:23:01 +03:00
Anna Shaleva
e46d76d7aa core: rename state.NEP17Balances to state.NEP17TransferInfo
Balances are to be removed from state.NEP17TransferInfo, so the remnant
fields are NextTransferBatch, NewBatch and a map of LastUpdatedBlocks.
These fields are more staff-related.

Also rename dao.[Get, Put, put]NEP17Balances and STNEP17Balances
preffix.

Also rename NEP17TransferInfo.Trackers to LastUpdatedBlockTrackers
because NEP17TransferInfo.Balances are to be removed.
2021-07-28 13:22:53 +03:00
Anna Shaleva
c0a2c74e0c core: maintain a set of NEP17-compliant contracts 2021-07-28 13:22:53 +03:00
Roman Khimov
50d99464e0
Merge pull request #2064 from nspcc-dev/fix-remove-stale-hang
mempool: send events in a separate goroutine
2021-07-23 18:16:14 +03:00
Evgeniy Stratonikov
e2f2addf95 notary: fix possible deadlock in UpdateNotaryNodes
`UpdateNotaryNodes` takes account then request mutex, and `PostPersist` takes
them in a different order. Because they are executed concurrently a deadlock
can appear.

```
2021-07-23T11:06:58.3732405Z panic: test timed out after 10m0s
2021-07-23T11:06:58.3732642Z
2021-07-23T11:06:58.3742610Z goroutine 7351 [semacquire, 9 minutes]:
2021-07-23T11:06:58.3743140Z sync.runtime_SemacquireMutex(0xc00010e4dc, 0x1100000000, 0x1)
2021-07-23T11:06:58.3743747Z 	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/sema.go:71 +0x47
2021-07-23T11:06:58.3744222Z sync.(*Mutex).lockSlow(0xc00010e4d8)
2021-07-23T11:06:58.3744742Z 	/opt/hostedtoolcache/go/1.14.15/x64/src/sync/mutex.go:138 +0x1c1
2021-07-23T11:06:58.3745209Z sync.(*Mutex).Lock(0xc00010e4d8)
2021-07-23T11:06:58.3745692Z 	/opt/hostedtoolcache/go/1.14.15/x64/src/sync/mutex.go:81 +0x7d
2021-07-23T11:06:58.3746162Z sync.(*RWMutex).Lock(0xc00010e4d8)
2021-07-23T11:06:58.3746764Z 	/opt/hostedtoolcache/go/1.14.15/x64/src/sync/rwmutex.go:98 +0x4a
2021-07-23T11:06:58.3747699Z github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).UpdateNotaryNodes(0xc00010e480, 0xc000105b90, 0x1, 0x1)
2021-07-23T11:06:58.3748621Z 	/home/runner/work/neo-go/neo-go/pkg/services/notary/node.go:44 +0x3ba
2021-07-23T11:06:58.3749367Z github.com/nspcc-dev/neo-go/pkg/core.TestNotary(0xc0003677a0)
2021-07-23T11:06:58.3750116Z 	/home/runner/work/neo-go/neo-go/pkg/core/notary_test.go:594 +0x2dba
2021-07-23T11:06:58.3750641Z testing.tRunner(0xc0003677a0, 0x16f3308)
2021-07-23T11:06:58.3751202Z 	/opt/hostedtoolcache/go/1.14.15/x64/src/testing/testing.go:1050 +0x1ec
2021-07-23T11:06:58.3751696Z created by testing.(*T).Run
2021-07-23T11:06:58.3752225Z 	/opt/hostedtoolcache/go/1.14.15/x64/src/testing/testing.go:1095 +0x538
2021-07-23T11:06:58.3752573Z
2021-07-23T11:06:58.3771319Z goroutine 7340 [semacquire, 9 minutes]:
2021-07-23T11:06:58.3772048Z sync.runtime_SemacquireMutex(0xc00010e504, 0x0, 0x0)
2021-07-23T11:06:58.3772889Z 	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/sema.go:71 +0x47
2021-07-23T11:06:58.3773581Z sync.(*RWMutex).RLock(0xc00010e4f8)
2021-07-23T11:06:58.3774310Z 	/opt/hostedtoolcache/go/1.14.15/x64/src/sync/rwmutex.go:50 +0xa4
2021-07-23T11:06:58.3775449Z github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).getAccount(0xc00010e480, 0x0)
2021-07-23T11:06:58.3776626Z 	/home/runner/work/neo-go/neo-go/pkg/services/notary/node.go:51 +0x51
2021-07-23T11:06:58.3778270Z github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).finalize(0xc00010e480, 0xc0003b2630, 0xa97df1bc78dd5787, 0xcc8a4d69e7f5d62a, 0x1a4d7981bd86b087, 0xbafdb720c93480b3, 0x0, 0x0)
2021-07-23T11:06:58.3779845Z 	/home/runner/work/neo-go/neo-go/pkg/services/notary/notary.go:306 +0x54
2021-07-23T11:06:58.3781022Z github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).PostPersist(0xc00010e480)
2021-07-23T11:06:58.3782232Z 	/home/runner/work/neo-go/neo-go/pkg/services/notary/notary.go:297 +0x662
2021-07-23T11:06:58.3782989Z github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).Run(0xc00010e480)
2021-07-23T11:06:58.3783941Z 	/home/runner/work/neo-go/neo-go/pkg/services/notary/notary.go:148 +0x3cb
2021-07-23T11:06:58.3784702Z created by github.com/nspcc-dev/neo-go/pkg/core.TestNotary
2021-07-23T11:06:58.3785451Z 	/home/runner/work/neo-go/neo-go/pkg/core/notary_test.go:132 +0x6e0
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-23 14:48:00 +03:00
Evgeniy Stratonikov
3507f52c32 notary: process new transactions in a separate goroutine
Related #2063.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-23 14:48:00 +03:00
Roman Khimov
6103da8d10 context: read item key in LE
Hi, neo-project/neo#938.
2021-07-23 12:43:04 +03:00
Roman Khimov
efb67a0ea3 context: scripts and signatures are base64-encoded in C# now
So use base64 too and add compatibility test. Unfortunately this breaks
support for old (hex-based) files, but those should be completed a long time
ago.
2021-07-23 11:57:35 +03:00
Roman Khimov
cbe1eeb08c smartcontract: add support for valueless Parameters
This is fine:
            {
               "type" : "Signature"
            },
2021-07-23 11:57:13 +03:00
Roman Khimov
59b4377f90 context: support Neo.Network.P2P.Payloads.Transaction type
C# now uses this one, so use it by default, but also accept old one.
2021-07-23 11:33:51 +03:00
Roman Khimov
ad35db66b5
Merge pull request #2091 from nspcc-dev/tune-error-messages
*: simplify some error messages
2021-07-23 10:40:26 +03:00
Roman Khimov
6e2eddbeb9
Merge pull request #2090 from nspcc-dev/new-query-commands
New query commands
2021-07-23 10:39:57 +03:00
Roman Khimov
1d8ad5b84a *: simplify some error messages
Log:
2021-07-23T09:59:18.948+0300    WARN    contract invocation failed      {"tx": "de3e3c1f1d37e4528990f894dea5583fd320485ad3862a95eb0e8823eecf4a5f", "block": 9643, "error": "error encountered at instruction 1 (SYSCALL): error during call from native: error encountered at instruction 745 (CAT): invalid conversion: Map/ByteString"}

The word "error" appears 4 times here.
2021-07-23 10:08:09 +03:00
Roman Khimov
7366d45985 rpc: add GetStateHeight to client 2021-07-22 21:13:44 +03:00
Roman Khimov
a188d20fd1 rpc: fix getstateheight result compatibility
C#:
   "result" : {
      "localrootindex" : 11623,
      "validatedrootindex" : 11623
   }

Go:
   "result" : {
      "blockHeight" : 11627,
      "stateHeight" : 11627
   }
2021-07-22 21:13:44 +03:00
Roman Khimov
6b852fc7b6
Merge pull request #2089 from nspcc-dev/fix-emit
vm/emit: improve error message
2021-07-22 15:17:59 +03:00
Evgeniy Stratonikov
808c30e7d7 vm/emit: improve error message
Show unsupported type instead of value.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-22 14:23:32 +03:00
Roman Khimov
ede410a4a7 go.mod: update ishell package
It only adds go.mod and changes import path, that's it.
2021-07-21 23:28:26 +03:00
Roman Khimov
002ad9dfee go.mod: update miniredis to 2.15.1
It's only used for testing purposes and this version doesn't change anything
for us, but still better be current.
2021-07-21 23:28:26 +03:00
Roman Khimov
4d1e952be6 go.mod: update go-datastructures to 1.0.53
We're only using queue library and it didn't change in any way, but 1.0.53 has
proper go.mod, so it's still an improvement.

It at the same time pulls some new packages also like x/tools.
2021-07-21 23:28:00 +03:00
Roman Khimov
4d2ecab16f consensus: fix nonce handling
It was broken somewhere between 2f490a3403 and
85ce207f40 leading to panic on watch only node:

2021-07-21T16:21:39.201+0200    INFO    received Commit {"validator": 3}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xbcc59e]

goroutine 486 [running]:
github.com/nspcc-dev/neo-go/pkg/consensus.(*service).newBlockFromContext(0xc0001629a0, 0xc000308000, 0xc0010fa000, 0x2cb417800)
        github.com/nspcc-dev/neo-go/pkg/consensus/consensus.go:664 +0xbe
github.com/nspcc-dev/dbft.(*Context).MakeHeader(...)
        github.com/nspcc-dev/dbft@v0.0.0-20210302103605-cc75991b7cfb/context.go:270
github.com/nspcc-dev/dbft.(*DBFT).onCommit(0xc000308000, 0x138c998, 0xc000115110)
        github.com/nspcc-dev/dbft@v0.0.0-20210302103605-cc75991b7cfb/dbft.go:487 +0x575
github.com/nspcc-dev/dbft.(*DBFT).OnReceive(0xc000308000, 0x138c998, 0xc000115110)
        github.com/nspcc-dev/dbft@v0.0.0-20210302103605-cc75991b7cfb/dbft.go:251 +0xef5
github.com/nspcc-dev/neo-go/pkg/consensus.(*service).eventLoop(0xc0001629a0)
        github.com/nspcc-dev/neo-go/pkg/consensus/consensus.go:312 +0x7d6
created by github.com/nspcc-dev/neo-go/pkg/consensus.(*service).Start
        github.com/nspcc-dev/neo-go/pkg/consensus/consensus.go:262 +0xdc

In fact, nonce is correctly provided by dbft library (since Legacy), we just
need to use it here.
2021-07-21 19:06:19 +03:00
Roman Khimov
7d6898677b keys: trivial code simplification 2021-07-21 17:05:49 +03:00
Roman Khimov
e3f19dd242
Merge pull request #2081 from nspcc-dev/mainnet-config-update
config: update mainnet magic
2021-07-21 15:03:58 +03:00
Roman Khimov
df07ba505a config: update mainnet magic
It's NEO3, see neo-project/neo-node#795.
2021-07-21 14:42:26 +03:00
Roman Khimov
5bdcd4c241 client: add GetCandidateRegisterPrice method
It's important for clients.
2021-07-21 12:19:55 +03:00
Roman Khimov
35c2c3ae8e
Merge pull request #2078 from nspcc-dev/configurable-initial-gas
config: add InitialGASSupply, fix #2073
2021-07-20 17:10:25 +03:00
Roman Khimov
36d486a664 config: add InitialGASSupply, fix #2073
We now have 52M by default.
2021-07-20 16:59:54 +03:00
Roman Khimov
caf07c1ee7
Merge pull request #2076 from nspcc-dev/fix-occasional-bolt-test-failures
Improve temp file/dir handling in tests
2021-07-20 16:53:54 +03:00
Roman Khimov
f9a9d15490 config: update testnet magic for RC4
See neo-project/neo-node#798 and https://github.com/neo-project/neo-node/releases/tag/v3.0.0-rc4
2021-07-20 13:16:38 +03:00
Roman Khimov
0583f252ab *: create real temporary dirs and files in tests
Improve reliability.
2021-07-20 12:51:11 +03:00
Roman Khimov
3b19b34122 storage: fix memcached test with boltdb store
Everything was wrong here, wrong file used, wrong cleanup procedure, the net
result is this (and some failing tests from time to time):

  $ ls -l /tmp/test_bolt_db* | wc -l
  30939
2021-07-20 12:35:24 +03:00
Roman Khimov
c88ebaede9
Merge pull request #2075 from nspcc-dev/small-refactoring
Array util refactoring and naming improvement
2021-07-20 11:29:59 +03:00
Roman Khimov
7477e2cd9f
Merge pull request #2074 from nspcc-dev/fix-oracle-service-behaviour
Fix oracle service behaviour
2021-07-20 11:29:39 +03:00
Roman Khimov
19717dd9a8 slice: introduce common Copy helper
It's a bit more convenient to use.
2021-07-19 22:57:55 +03:00
Roman Khimov
a54e3516d1 slice: add Reverse function, deduplicate code a bit 2021-07-19 22:57:55 +03:00
Roman Khimov
100e97d772 util: move ArrayReverse into package of its own
Leave just uint160 and uint256 types in util.
2021-07-19 22:57:55 +03:00
Roman Khimov
3a93977b7b oracle: only process new requests after initial sync
If an oracle node is resynchronized from the genesis the service receives all
requests from all blocks via AddRequests() invoked from the native
contract. Almost all of them are long obsolete and need to be removed, native
oracle contract will try to do that with RemoveRequests() calls, but they
won't change anything.

So queue up all "initial" requests in special map and manage it directly
before the module is Run() which happens after synchronization
completion. Then process any requests that are still active and work with new
blocks as usual.
2021-07-19 22:52:59 +03:00
Roman Khimov
588f3fbbd3 state: drop State from NEOBalance and NEP17Balance
We're in the `state` package already.
2021-07-19 22:01:07 +03:00
Roman Khimov
f4a9139a05
Merge pull request #2071 from nspcc-dev/fix-more-vm-bugs
Fix more VM bugs
2021-07-19 22:00:17 +03:00
Roman Khimov
0bad413e23 oracle: ensure proper ValidUntilBlock for backup tx
See neo-project/neo-modules#608.
2021-07-19 21:58:10 +03:00
Roman Khimov
1568ebc513 stackitem: rework struct cloning protection
Count everything, fail early, make it more compatible with
neo-project/neo-vm#423.
2021-07-19 15:42:42 +03:00
Roman Khimov
79b1bf72aa native: check that oracle request GAS IsInt64()
We use int64 value down below, so check for IsInt64() while MinimumResponseGas
check still covers for values less than zero.
2021-07-19 15:42:42 +03:00
Roman Khimov
5f13de3a76 *: simplify some integer checks with IsUint64()
If we need a positive number we can do `IsUint64()` instead of checking that
`Int64()` is `> 0`.
2021-07-19 15:42:42 +03:00
Roman Khimov
233307aca5 stackitem: completely drop MaxArraySize
Turns out C# VM doesn't have it since preview2, so our limiting of
MaxArraySize in incompatible with it. Removing this limit shouldn't be a
problem with the reference counter we have, both APPEND and SETITEM add things
to reference counter and we can't exceed MaxStackSize. PACK on the other hand
can't get more than MaxStackSize-1 of input elements.

Unify NEWSTRUCT with NEWARRAY* and use better integer checks at the same time.

Multisig limit is still 1024.
2021-07-19 15:42:42 +03:00
Roman Khimov
ee4a647934 stackitem: limit EQUAL for deeply nested structs
See neo-project/neo-vm#428.
2021-07-19 15:42:42 +03:00
Roman Khimov
f89f0300f6 stackitem: improve test coverage a bit 2021-07-19 15:42:42 +03:00
Roman Khimov
df2430d5e4 stackitem: limit deserialization to MaxDeserialized items
Follow neo-project/neo#2531. Even though it's not strictly required (our node
handles problematic script just fine) we better be compliant wrt
deserialization behavior. MaxDeserialized is introduced to avoid moving
MaxStackSize which is a VM parameter.
2021-07-19 15:42:42 +03:00
Roman Khimov
aab18c3083 stackitem: introduce Convertible interface
We have a lot of native contract types that are converted to stack items
before serialization, then deserialized as stack items and converted back to
regular structures. stackitem.Convertible allows to remove a lot of repetitive
io.Serializable code.

This also introduces to/from converter in testserdes which unfortunately
required to change util tests to avoid circular references.
2021-07-19 15:42:42 +03:00
Roman Khimov
2d993d0da5 state: store notary deposit as stackitem
Which is less effective, but makes it more similar to other native contracts
that are supposed to be contracts anyway.
2021-07-19 15:42:42 +03:00
Roman Khimov
70ddbf7180 native: reuse stackitem.(De)Serialize more for data structures
Less code bloat, no functional changes.
2021-07-19 15:42:42 +03:00
Roman Khimov
4775b513f9 native: do proper error handling when deserializing user data 2021-07-19 15:42:42 +03:00
Roman Khimov
fbe8bd2d9c stackitem: limit deserialized arrays/maps
See neo-project/neo#2531, even though they use MaxStackSize there resulting
element is not valid unless it has <=MaxArraySize elements.
2021-07-19 15:42:42 +03:00
Roman Khimov
654c4a6589 stackitem: properly pass allowInvalid on binary deserialization
Otherwise DecodeBinaryProtected() is broken for arrays/maps.
2021-07-19 15:42:41 +03:00
Roman Khimov
3a991abb62 fee: adjust SQRT price
See neo-project/neo#2540.
2021-07-19 15:42:41 +03:00
Roman Khimov
15be763bb3 vm: limit POW, fix #2060
Calculating pow(pow(2, 255), 0xffffffff) takes unknown amount of time. See
also neo-project/neo-vm#422.
2021-07-19 15:42:41 +03:00
Roman Khimov
5e26170b79
Merge pull request #2072 from nspcc-dev/address-as-account-in-rpc-params
rpc/request: support passing accounts as addresses
2021-07-19 10:09:15 +03:00
Roman Khimov
a617390185 rpc/request: support passing accounts as addresses
Turns out, it's completely legal in C#, see neo-project/neo#2543.
2021-07-18 16:45:53 +03:00
Roman Khimov
21e05f5779
Merge pull request #2055 from nspcc-dev/fix-gas-limits
Tune fixed GAS limits
2021-07-16 16:53:06 +03:00
Evgeniy Stratonikov
85ce207f40 consensus: add Nonce to PrepareRequest
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-15 16:00:02 +03:00
Evgeniy Stratonikov
8077f9232d interop: implement System.Runtime.GetRandom
Our murmur3 implementation is architecture independent and optimized in
assembly.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-15 16:00:01 +03:00
Evgeniy Stratonikov
fdb54f2dc3 core/test: get rid of empty tx scripts
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-15 15:58:49 +03:00
Evgeniy Stratonikov
9c5b4e5916 encoding/bigint: allow to convert unsigned integers
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-15 15:58:49 +03:00
Evgeniy Stratonikov
3a4e0caeb8 core/block: add Nonce field to header
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-15 15:58:49 +03:00
Evgeniy Stratonikov
2cd5b63f0b policy: fix max exec fee
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-14 10:27:09 +03:00
Evgeniy Stratonikov
451b02122a *: increase GAS for verification
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-14 10:27:09 +03:00
Roman Khimov
2b7abd20e7
Merge pull request #2057 from nspcc-dev/optimize-storage-find
Optimize `storageFind`
2021-07-13 12:51:10 +03:00
Roman Khimov
83a557f0eb
Merge pull request #2042 from nspcc-dev/oracle-not-supported
Check oracle response Content-Type header
2021-07-13 11:31:37 +03:00
Roman Khimov
d638aee7a1
Merge pull request #2053 from nspcc-dev/stackitem-json
Speedup stackitem json serialization
2021-07-13 11:27:33 +03:00
Evgeniy Stratonikov
0611031fd4 stackitem/test: improve coverage for serialization
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-13 11:06:19 +03:00
Evgeniy Stratonikov
8fdc7e32f5 stackitem: cache serialized results in EncodeBinary
```
name            old time/op    new time/op    delta
EncodeBinary-8    8.39ms ± 1%    0.05ms ±21%  -99.44%  (p=0.000 n=10+9)

name            old alloc/op   new alloc/op   delta
EncodeBinary-8    2.24MB ± 0%    0.33MB ± 0%  -85.29%  (p=0.000 n=9+10)

name            old allocs/op  new allocs/op  delta
EncodeBinary-8     65.6k ± 0%      0.0k ± 0%  -99.95%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-13 11:06:19 +03:00
Evgeniy Stratonikov
17a3f17c74 stackitem: use byte-slice directly during encoding
```
name            old time/op    new time/op    delta
EncodeBinary-8    10.6ms ± 1%     8.4ms ± 1%  -20.94%  (p=0.000 n=9+10)

name            old alloc/op   new alloc/op   delta
EncodeBinary-8    1.64MB ± 0%    2.24MB ± 0%  +36.18%  (p=0.000 n=8+9)

name            old allocs/op  new allocs/op  delta
EncodeBinary-8      131k ± 0%       66k ± 0%  -49.98%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-13 11:06:18 +03:00
Evgeniy Stratonikov
c4eb7db8a5 stackitem: access value of Array/Struct directly
```
name      old time/op    new time/op    delta
ToJSON-8    50.3µs ±34%    47.8µs ± 9%     ~     (p=0.971 n=10+10)

name      old alloc/op   new alloc/op   delta
ToJSON-8     396kB ± 0%     396kB ± 0%   -0.10%  (p=0.000 n=6+10)

name      old allocs/op  new allocs/op  delta
ToJSON-8      34.0 ± 0%      18.0 ± 0%  -47.06%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:40:21 +03:00
Evgeniy Stratonikov
dc0a17dd0e stackitem: cache visited items while marshaling json
```
name      old time/op    new time/op    delta
ToJSON-8    4.52ms ± 4%    0.05ms ±34%  -98.89%  (p=0.000 n=8+10)

name      old alloc/op   new alloc/op   delta
ToJSON-8    2.13MB ± 0%    0.40MB ± 0%  -81.44%  (p=0.000 n=9+6)

name      old allocs/op  new allocs/op  delta
ToJSON-8     65.6k ± 0%      0.0k ± 0%  -99.95%  (p=0.000 n=10+10)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:40:21 +03:00
Evgeniy Stratonikov
69cdd5252a stackitem: add benchmark for serialization routines
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:40:17 +03:00
Evgeniy Stratonikov
eb7bd7b99b core: optimize storageFind
Because `Map` stores elements in arbitrary order, addition of new
element takes linear time (`Index` iterates over all keys). Thus our
`storageFind` is actually quadratic in time. Optimize this by creating
map from sorted slice.

```
name           old time/op    new time/op    delta
StorageFind-8     157µs ± 2%     112µs ± 1%  -28.60%  (p=0.000 n=10+10)

name           old alloc/op   new alloc/op   delta
StorageFind-8    69.4kB ± 0%    60.5kB ± 0%  -12.90%  (p=0.000 n=9+10)

name           old allocs/op  new allocs/op  delta
StorageFind-8     2.21k ± 0%     2.00k ± 0%   -9.37%  (p=0.000 n=10+7)
```

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:22:17 +03:00
Evgeniy Stratonikov
b8b272c8c4 core/test: add benchmark for storageFind
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:20:45 +03:00
Evgeniy Stratonikov
0d5ede0bff core/test: use testing.TB in constructors
This allows to use chain in benchmarks.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:20:45 +03:00
Evgeniy Stratonikov
9f377cde12 storage: convert key once in MemoryStore.seek
There is no need in additional allocations.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 14:14:32 +03:00
Evgeniy Stratonikov
e936e5e295 oracle: set request Content-Type
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 13:13:48 +03:00
Evgeniy Stratonikov
8e9302f40b oracle: check response Content-Type
If not specified everything is allowed.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-12 13:13:48 +03:00
Roman Khimov
9de5c850e7
Merge pull request #2054 from nspcc-dev/fix-oom-on-struct-cloning
Fix OOM on struct cloning
2021-07-12 13:04:12 +03:00
Roman Khimov
fe3c68b92d vm: panic in cloneIfStruct
It's internal, so we can deduplicate code a bit.
2021-07-12 11:52:24 +03:00
Roman Khimov
bc5a1b7bcd
Merge pull request #2043 from nspcc-dev/runtime-getnetwork
interop/runtime: add `System.Runtime.GetNetwork`, fix #2038
2021-07-12 10:46:41 +03:00
Roman Khimov
4000dd692c vm: make cloning limits a bit more effective
Take actual reference counter value into account.
2021-07-11 19:37:06 +03:00
Roman Khimov
cfe41abd35 vm: fix OOM during nested structure cloning
Resulting item can't have more than MaxStackSize elements. Technically this
limits to MaxStackSize cloned elements but it's considered to be enough to
mitigate the issue (the next size check is going to happen during push to the
stack). See neo-project/neo#2534, thanks @vang1ong7ang.
2021-07-11 18:47:50 +03:00
Roman Khimov
1853d0c713 core: don't copy config in ApplyPolicyToTxSet
Makes no sense, GetConfig() is for external users.
2021-07-09 13:01:42 +03:00
Roman Khimov
fc83b34d02
Merge pull request #2052 from nspcc-dev/fix-state-276
vm: copy integers for unary operations, fix #2051
2021-07-08 17:30:02 +03:00
Evgeniy Stratonikov
1863023e27 vm: copy argument in unary opcodes, fix #2051
We already do this in binary opcodes, let's be consistent.

This fixes state difference at height 275663
for tx 3c498317684d63849b03e4c58ad57ce4b19bb206b7b01bcc64233de3b3e207f4

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-08 17:09:04 +03:00
Evgeniy Stratonikov
96a42cd011 interop: add System.Runtime.GetNetwork, fix #2038
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-08 17:02:37 +03:00
Roman Khimov
7bc7457fbe
Merge pull request #2050 from nspcc-dev/fix-network-test-race
Fix some test failures
2021-07-08 14:50:50 +03:00
Roman Khimov
b8192d0958 network: optimize waiting in test
require.Eventually polls more often reducing average waiting time.
2021-07-08 11:14:35 +03:00
Roman Khimov
9cc4f42a71 network: fix discoverer test
Asynchronous tryAddress() routines may get dial result AFTER the switch to
another test, so we need to ensure that they'll get the result intended for
this particular call. Fixes:

2021-07-07T20:25:40.1624521Z === RUN   TestDefaultDiscoverer
2021-07-07T20:25:40.1625316Z     discovery_test.go:159: timeout expecting for transport dial; i: 2, j: 1
2021-07-07T20:25:40.1626319Z --- FAIL: TestDefaultDiscoverer (1.19s)
2021-07-08 11:03:30 +03:00
Roman Khimov
bd400dfe20 native: copy contract on update
It can be a pointer to cached data:

2021-07-07T20:05:24.8508647Z ==================
2021-07-07T20:05:24.8509294Z WARNING: DATA RACE
2021-07-07T20:05:24.8510243Z Write at 0x00c000142cc8 by goroutine 51:
2021-07-07T20:05:24.8511350Z   github.com/nspcc-dev/neo-go/pkg/core/native.(*Management).Update()
2021-07-07T20:05:24.8529622Z       /home/runner/work/neo-go/neo-go/pkg/core/native/management.go:335 +0x70a
2021-07-07T20:05:24.8531266Z   github.com/nspcc-dev/neo-go/pkg/core/native.(*Management).updateWithData()
2021-07-07T20:05:24.8532544Z       /home/runner/work/neo-go/neo-go/pkg/core/native/management.go:316 +0x1e9
2021-07-07T20:05:24.8533726Z   github.com/nspcc-dev/neo-go/pkg/core/native.(*Management).update()
2021-07-07T20:05:24.8534928Z       /home/runner/work/neo-go/neo-go/pkg/core/native/management.go:303 +0x111
2021-07-07T20:05:24.8536123Z   github.com/nspcc-dev/neo-go/pkg/core/native.(*Management).update-fm()
2021-07-07T20:05:24.8537389Z       /home/runner/work/neo-go/neo-go/pkg/core/native/management.go:302 +0x34
2021-07-07T20:05:24.8538462Z   github.com/nspcc-dev/neo-go/pkg/core/native.Call()
2021-07-07T20:05:24.8539565Z       /home/runner/work/neo-go/neo-go/pkg/core/native/interop.go:54 +0x90c
2021-07-07T20:05:24.8540751Z   github.com/nspcc-dev/neo-go/pkg/core/interop.(*Context).SyscallHandler()
2021-07-07T20:05:24.8542177Z       /home/runner/work/neo-go/neo-go/pkg/core/interop/context.go:262 +0x216
2021-07-07T20:05:24.8543328Z   github.com/nspcc-dev/neo-go/pkg/core/interop.(*Context).SyscallHandler-fm()
2021-07-07T20:05:24.8544586Z       /home/runner/work/neo-go/neo-go/pkg/core/interop/context.go:250 +0x53
2021-07-07T20:05:24.8545648Z   github.com/nspcc-dev/neo-go/pkg/vm.(*VM).execute()
2021-07-07T20:05:24.8546635Z       /home/runner/work/neo-go/neo-go/pkg/vm/vm.go:1312 +0xb2d5
2021-07-07T20:05:24.8547632Z   github.com/nspcc-dev/neo-go/pkg/vm.(*VM).Step()
2021-07-07T20:05:24.8548640Z       /home/runner/work/neo-go/neo-go/pkg/vm/vm.go:416 +0x1f2
2021-07-07T20:05:24.8549836Z   github.com/nspcc-dev/neo-go/pkg/vm.(*VM).Run()
2021-07-07T20:05:24.8550852Z       /home/runner/work/neo-go/neo-go/pkg/vm/vm.go:393 +0x195
2021-07-07T20:05:24.8551989Z   github.com/nspcc-dev/neo-go/pkg/rpc/server.(*Server).runScriptInVM()
2021-07-07T20:05:24.8553174Z       /home/runner/work/neo-go/neo-go/pkg/rpc/server/server.go:1326 +0x7eb
2021-07-07T20:05:24.8554345Z   github.com/nspcc-dev/neo-go/pkg/rpc/server.(*Server).invokeFunction()
2021-07-07T20:05:24.8555560Z       /home/runner/work/neo-go/neo-go/pkg/rpc/server/server.go:1219 +0x50c
2021-07-07T20:05:24.8556616Z   github.com/nspcc-dev/neo-go/pkg/rpc/server.(*Server).handleIn()
2021-07-07T20:05:24.8557823Z       /home/runner/work/neo-go/neo-go/pkg/rpc/server/server.go:345 +0xecb
2021-07-07T20:05:24.8558987Z   github.com/nspcc-dev/neo-go/pkg/rpc/server.(*Server).handleRequest()
2021-07-07T20:05:24.8560356Z       /home/runner/work/neo-go/neo-go/pkg/rpc/server/server.go:315 +0x364
2021-07-07T20:05:24.8561599Z   github.com/nspcc-dev/neo-go/pkg/rpc/server.(*Server).handleHTTPRequest()
2021-07-07T20:05:24.8562976Z       /home/runner/work/neo-go/neo-go/pkg/rpc/server/server.go:309 +0xf9b
2021-07-07T20:05:24.8564208Z   github.com/nspcc-dev/neo-go/pkg/rpc/server.(*Server).handleHTTPRequest-fm()
2021-07-07T20:05:24.8565523Z       /home/runner/work/neo-go/neo-go/pkg/rpc/server/server.go:257 +0x64
2021-07-07T20:05:24.8566348Z   net/http.HandlerFunc.ServeHTTP()
2021-07-07T20:05:24.8567290Z       /opt/hostedtoolcache/go/1.15.13/x64/src/net/http/server.go:2042 +0x51
2021-07-07T20:05:24.8568185Z   net/http.serverHandler.ServeHTTP()
2021-07-07T20:05:24.8569130Z       /opt/hostedtoolcache/go/1.15.13/x64/src/net/http/server.go:2843 +0xca
2021-07-07T20:05:24.8569861Z   net/http.(*conn).serve()
2021-07-07T20:05:24.8570634Z       /opt/hostedtoolcache/go/1.15.13/x64/src/net/http/server.go:1925 +0x84c
2021-07-07T20:05:24.8571139Z
2021-07-07T20:05:24.8571639Z Previous read at 0x00c000142cc8 by goroutine 41:
2021-07-07T20:05:24.8572892Z   github.com/nspcc-dev/neo-go/pkg/core/interop/contract.callExFromNative()
2021-07-07T20:05:24.8574265Z       /home/runner/work/neo-go/neo-go/pkg/core/interop/contract/call.go:101 +0x2d4
2021-07-07T20:05:24.8575581Z   github.com/nspcc-dev/neo-go/pkg/core/interop/contract.callInternal()
2021-07-07T20:05:24.8576951Z       /home/runner/work/neo-go/neo-go/pkg/core/interop/contract/call.go:85 +0x1f5
2021-07-07T20:05:24.8578094Z   github.com/nspcc-dev/neo-go/pkg/core/interop/contract.Call()
2021-07-07T20:05:24.8579306Z       /home/runner/work/neo-go/neo-go/pkg/core/interop/contract/call.go:69 +0x804
2021-07-07T20:05:24.8580514Z   github.com/nspcc-dev/neo-go/pkg/core/interop.(*Context).SyscallHandler()
2021-07-07T20:05:24.8581755Z       /home/runner/work/neo-go/neo-go/pkg/core/interop/context.go:262 +0x216
2021-07-07T20:05:24.8582907Z   github.com/nspcc-dev/neo-go/pkg/core/interop.(*Context).SyscallHandler-fm()
2021-07-07T20:05:24.8584300Z       /home/runner/work/neo-go/neo-go/pkg/core/interop/context.go:250 +0x53
2021-07-07T20:05:24.8585422Z   github.com/nspcc-dev/neo-go/pkg/vm.(*VM).execute()
2021-07-07T20:05:24.8586403Z       /home/runner/work/neo-go/neo-go/pkg/vm/vm.go:1312 +0xb2d5
2021-07-07T20:05:24.8590451Z   github.com/nspcc-dev/neo-go/pkg/vm.(*VM).Step()
2021-07-07T20:05:24.8591549Z       /home/runner/work/neo-go/neo-go/pkg/vm/vm.go:416 +0x1f2
2021-07-07T20:05:24.8592466Z   github.com/nspcc-dev/neo-go/pkg/vm.(*VM).Run()
2021-07-07T20:05:24.8593550Z       /home/runner/work/neo-go/neo-go/pkg/vm/vm.go:393 +0x195
2021-07-07T20:05:24.8594531Z   github.com/nspcc-dev/neo-go/pkg/core.(*Blockchain).storeBlock()
2021-07-07T20:05:24.8595708Z       /home/runner/work/neo-go/neo-go/pkg/core/blockchain.go:717 +0x1694
2021-07-07T20:05:24.8596793Z   github.com/nspcc-dev/neo-go/pkg/core.(*Blockchain).AddBlock()
2021-07-07T20:05:24.8598142Z       /home/runner/work/neo-go/neo-go/pkg/core/blockchain.go:579 +0x544
2021-07-07T20:05:24.8599419Z   github.com/nspcc-dev/neo-go/pkg/consensus.(*service).processBlock()
2021-07-07T20:05:24.8601093Z       /home/runner/work/neo-go/neo-go/pkg/consensus/consensus.go:521 +0x141
2021-07-07T20:05:24.8602485Z   github.com/nspcc-dev/neo-go/pkg/consensus.(*service).processBlock-fm()
2021-07-07T20:05:24.8603809Z       /home/runner/work/neo-go/neo-go/pkg/consensus/consensus.go:517 +0x55
2021-07-07T20:05:24.8604910Z   github.com/nspcc-dev/dbft.(*DBFT).checkCommit()
2021-07-07T20:05:24.8606098Z       /home/runner/go/pkg/mod/github.com/nspcc-dev/dbft@v0.0.0-20210302103605-cc75991b7cfb/check.go:73 +0xd8a
2021-07-07T20:05:24.8607235Z   github.com/nspcc-dev/dbft.(*DBFT).checkPrepare()
2021-07-07T20:05:24.8608551Z       /home/runner/go/pkg/mod/github.com/nspcc-dev/dbft@v0.0.0-20210302103605-cc75991b7cfb/check.go:36 +0x677
2021-07-07T20:05:24.8609681Z   github.com/nspcc-dev/dbft.(*DBFT).sendPrepareRequest()
2021-07-07T20:05:24.8611016Z       /home/runner/go/pkg/mod/github.com/nspcc-dev/dbft@v0.0.0-20210302103605-cc75991b7cfb/send.go:42 +0x47c
2021-07-07T20:05:24.8612107Z   github.com/nspcc-dev/dbft.(*DBFT).OnTimeout()
2021-07-07T20:05:24.8613257Z       /home/runner/go/pkg/mod/github.com/nspcc-dev/dbft@v0.0.0-20210302103605-cc75991b7cfb/dbft.go:188 +0x68c
2021-07-07T20:05:24.8614499Z   github.com/nspcc-dev/neo-go/pkg/consensus.(*service).eventLoop()
2021-07-07T20:05:24.8615712Z       /home/runner/work/neo-go/neo-go/pkg/consensus/consensus.go:274 +0x17ed
2021-07-07T20:05:24.8616284Z
2021-07-07T20:05:24.8616757Z Goroutine 51 (running) created at:
2021-07-07T20:05:24.8617367Z   net/http.(*Server).Serve()
2021-07-07T20:05:24.8618159Z       /opt/hostedtoolcache/go/1.15.13/x64/src/net/http/server.go:2969 +0x5d3
2021-07-07T20:05:24.8619253Z   github.com/nspcc-dev/neo-go/pkg/rpc/server.(*Server).Start.func2()
2021-07-07T20:05:24.8620437Z       /home/runner/work/neo-go/neo-go/pkg/rpc/server/server.go:224 +0x85
2021-07-07T20:05:24.8620936Z
2021-07-07T20:05:24.8621405Z Goroutine 41 (running) created at:
2021-07-07T20:05:24.8622393Z   github.com/nspcc-dev/neo-go/pkg/consensus.(*service).Start()
2021-07-07T20:05:24.8623562Z       /home/runner/work/neo-go/neo-go/pkg/consensus/consensus.go:250 +0x17c
2021-07-07T20:05:24.8624682Z   github.com/nspcc-dev/neo-go/pkg/network.(*Server).tryStartServices()
2021-07-07T20:05:24.8626078Z       /home/runner/work/neo-go/neo-go/pkg/network/server.go:445 +0x2e1
2021-07-07T20:05:24.8627226Z   github.com/nspcc-dev/neo-go/pkg/network.(*Server).Start()
2021-07-07T20:05:24.8628333Z       /home/runner/work/neo-go/neo-go/pkg/network/server.go:268 +0x2af
2021-07-07T20:05:24.8628973Z ==================
2021-07-08 10:40:54 +03:00
Roman Khimov
685ff8bbf6 network: reduce dial timeout in discoverer test
We don't care much about dialing, but the same constant is used in outer
discoverer loop in case no connections are established and we have no
connections established.
2021-07-08 10:40:54 +03:00
Roman Khimov
473a1fc24e rpc/server: fix sporadic test failures
It can still be "reading" after test end:

    subscription_test.go:24:
        	Error Trace:	subscription_test.go:24
        	            				asm_amd64.s:1374
        	Error:      	Received unexpected error:
        	            	set tcp 127.0.0.1:49874: use of closed network connection
        	Test:       	TestFilteredNotaryRequestSubscriptions

==================
WARNING: DATA RACE
Write at 0x00c000e28989 by goroutine 185:
  testing.tRunner()
      /opt/hostedtoolcache/go/1.15.13/x64/src/testing/testing.go:1126 +0x21a

Previous write at 0x00c000e28989 by goroutine 192:
  testing.(*common).FailNow()
      /opt/hostedtoolcache/go/1.15.13/x64/src/testing/testing.go:731 +0x4f
  testing.(*T).FailNow()
      <autogenerated>:1 +0x44
  github.com/stretchr/testify/require.NoError()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.6.1/require/require.go:1038 +0x104
  github.com/nspcc-dev/neo-go/pkg/rpc/server.wsReader()
      /home/runner/work/neo-go/neo-go/pkg/rpc/server/subscription_test.go:24 +0x187

Goroutine 185 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.15.13/x64/src/testing/testing.go:1168 +0x5bb
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.15.13/x64/src/testing/testing.go:1439 +0xa6
  testing.tRunner()
      /opt/hostedtoolcache/go/1.15.13/x64/src/testing/testing.go:1123 +0x202
  testing.runTests()
      /opt/hostedtoolcache/go/1.15.13/x64/src/testing/testing.go:1437 +0x612
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.15.13/x64/src/testing/testing.go:1345 +0x3b3
  main.main()
      _testmain.go:89 +0x236

Goroutine 192 (finished) created at:
  github.com/nspcc-dev/neo-go/pkg/rpc/server.initCleanServerAndWSClient()
      /home/runner/work/neo-go/neo-go/pkg/rpc/server/subscription_test.go:65 +0x294
  github.com/nspcc-dev/neo-go/pkg/rpc/server.TestFilteredNotaryRequestSubscriptions()
      /home/runner/work/neo-go/neo-go/pkg/rpc/server/subscription_test.go:348 +0x68d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.15.13/x64/src/testing/testing.go:1123 +0x202
==================
    testing.go:1038: race detected during execution of test
2021-07-08 10:40:54 +03:00
Roman Khimov
c2d444ace7 network: don't log transport errors on exit
Fix data race

==================
WARNING: DATA RACE
Read at 0x00c00012cca3 by goroutine 208:
  testing.(*common).logDepth()
      /usr/local/go/src/testing/testing.go:727 +0xa4
  testing.(*common).log()
      /usr/local/go/src/testing/testing.go:720 +0x8f
  testing.(*common).Logf()
      /usr/local/go/src/testing/testing.go:766 +0x21
  testing.(*T).Logf()
      <autogenerated>:1 +0x75
  go.uber.org/zap/zaptest.testingWriter.Write()
      /go/pkg/mod/go.uber.org/zap@v1.10.0/zaptest/logger.go:130 +0x11f
  go.uber.org/zap/zaptest.(*testingWriter).Write()
      <autogenerated>:1 +0xa9
  go.uber.org/zap/zapcore.(*ioCore).Write()
      /go/pkg/mod/go.uber.org/zap@v1.10.0/zapcore/core.go:90 +0x1c3
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      /go/pkg/mod/go.uber.org/zap@v1.10.0/zapcore/entry.go:215 +0x1e7
  go.uber.org/zap.(*Logger).Warn()
      /go/pkg/mod/go.uber.org/zap@v1.10.0/logger.go:195 +0x95
  github.com/nspcc-dev/neo-go/pkg/network.(*TCPTransport).Accept()
      /go/src/github.com/nspcc-dev/neo-go/pkg/network/tcp_transport.go:59 +0x8da

Previous write at 0x00c00012cca3 by goroutine 168:
  testing.tRunner.func1()
      /usr/local/go/src/testing/testing.go:1036 +0x467
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1054 +0x20d

Goroutine 208 (running) created at:
  github.com/nspcc-dev/neo-go/pkg/network.(*Server).Start()
      /go/src/github.com/nspcc-dev/neo-go/pkg/network/server.go:274 +0x391

Goroutine 168 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1095 +0x537
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1339 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1050 +0x1eb
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1337 +0x594
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1252 +0x2ff
  main.main()
      _testmain.go:90 +0x223
==================
2021-07-08 10:40:54 +03:00
Roman Khimov
ea9dde257c
Merge pull request #2047 from nspcc-dev/fix-state
Don't create an extension node for the last branch child
2021-07-08 10:20:59 +03:00
Roman Khimov
23fb5dd420 rpc/server: properly shutdown things in subscription test
Chain is the last one to Close().
2021-07-07 20:47:34 +03:00
Roman Khimov
f254cdf2db
Merge pull request #2046 from nspcc-dev/oracle-null-filter
jsonpath: add tests for `null` objects, fix #2039
2021-07-07 18:08:58 +03:00
Evgeniy Stratonikov
a33e422b79 mpt: do not create an Extension for the last child of a Branch
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-07 15:49:53 +03:00
Evgeniy Stratonikov
25428d53f1 mpt: strip branch node in a simple implementation
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-07 15:47:07 +03:00
Evgeniy Stratonikov
a4c7b3d0cf jsonpath: add tests for null objects, fix #2039
Turns out, nothing should be changed in our implementation because
nil-check is done implicitly by type casts or type switches.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-07 12:27:10 +03:00
Roman Khimov
0cd9cd0c80 state/result: save/return real JSONization errors
Don't hide/obfuscate real problems.
2021-07-07 00:42:36 +03:00
Roman Khimov
e62a766058 state: move nil check down to stackitem JSON processing
We want to return real errors, not some generic thing for any kind of thing
happening.
2021-07-07 00:38:19 +03:00
Roman Khimov
5a9efcc654 stackitem: rework error handling
Return good named errors everywhere, export appropriate constants, make
errors.Is() work.
2021-07-07 00:18:00 +03:00
Roman Khimov
0de949b575 stackitem: remove Item/StackItem from function names
They're useless in a package named 'stackitem', make this package a bit more
user-friendly.
2021-07-06 19:56:23 +03:00
Roman Khimov
b9ff07f32c stackitem: add limit to serialized items
Standard binary serialization/deserialization is mostly used in VM to put/get
elements into/from storage, so they never should exceed MaxSize (otherwise one
won't be able to deserialize these items).

This patch leaves EncodeBinaryStackItem unprotected, but that's a streaming
interface, so it's up to the user of it to ensure its appropriate use (and our
uses are mostly for native contract's data, so they're fine).
2021-07-06 19:34:02 +03:00
Roman Khimov
8472064bbc stackitem: refactor serialization code, add explicit context 2021-07-06 19:32:52 +03:00
Roman Khimov
1b7b7e4bec stackitem: don't overwrite error when serializing Item
We must get out as quickly as possible.
2021-07-06 17:51:46 +03:00
Roman Khimov
e34fa2e915 stackitem: limit JSONization nesting level
We can have very deep reference types and attempt to JSONize them can easily
lead to OOM (even though there are no recursive references inside). Therefore
we have to limit them. While regular ToJSON() is buffer size limited to
MaxSize, ToJSONWithTypes is not and limiting it to MaxSize output will require
substantial rewriting effort while not really providing fair result, MaxSize
is more about stack item size while its JSON representation can be much bigger
because of various overheads.

Initial thought was to limit it by element count based on
MaxIteratorResultItems, but the problem here is that we don't always have this
limit in contexts using ToJSONWithTypes (like notification event
marshaling). Thus we need some generic limit which would be fine for all
users.

We at the same time have maxJSONDepth used when deserializing from JSON, so
it can be used for marshaling as well, it's not often that we have deeper
structures in real results.

Inspired by neo-project/neo#2521.
2021-07-06 17:33:16 +03:00
Roman Khimov
6879cbcdcc stackitem: properly detect recursive references during serialization
If we're done with element it no longer can lead to recursion error, so fix
cases like `[arr, arr]` where we have two copies of `arr` trigger this error
for no good reason (there is no recursion there).
2021-07-06 17:10:31 +03:00
Roman Khimov
ae8f4ebd5e
Merge pull request #2034 from nspcc-dev/fix-state
Fix MPT branch node strip after deletion
2021-07-06 14:05:18 +03:00
Roman Khimov
ac126a300b callflag: add C#-compliant JSONization, fix #2040 2021-07-06 11:17:12 +03:00
Evgeniy Stratonikov
9691eee10c mpt: strip branch if 1 child is left
If the child left is a hash node, we should retrieve it from store.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-05 11:04:20 +03:00
Evgeniy Stratonikov
d2e8254848 native: fix typo in comments
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-05 11:04:19 +03:00
Evgeniy Stratonikov
a7722f0fa5 native: save balance value in storage for failed transfers
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-05 11:03:42 +03:00
Roman Khimov
e3323faf69
Merge pull request #2032 from nspcc-dev/optimize-restore
native: cache GAS per vote values
2021-07-01 15:17:37 +03:00
Evgeniy Stratonikov
0591366f85 native: cache GAS per vote values
`storage.Seek()` is rather expensive and we need only last updated value
of gas per block in `PostPersist()`.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-07-01 10:34:04 +03:00
Roman Khimov
c203995a9b
Merge pull request #2028 from nspcc-dev/core/fix-add-headers
core: allow to add several headers with StateRootInHeader on
2021-06-30 17:11:54 +03:00
Anna Shaleva
9673a04009 core: allow to add several headers with StateRootInHeader on
Problem: with StateRootInHeader setting on only one header of height N+1
can be added to the chain of height N, because we need local stateroot
to verify headers (which is calculated for the last stored block N).
Thus, adding chunk of headers starting from the current chain's heigh
is impossible and (*Blockchain).AddHeaders doesn't have much sense.

Solution: verify header.PrevStateRoot only for header N+1. Rest of the
headers should be added without PrevStateRoot verification.
2021-06-30 11:56:05 +03:00
Evgeniy Stratonikov
c9e6350915 manifest: fix group permission handling
We need to have at least one matching group in the manifest.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-06-30 11:02:45 +03:00
Evgeniy Stratonikov
4283e1003f manifest: fix wildcard permission handling
Wildcard contract can coexist with restricted set of methods.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-06-30 11:00:06 +03:00
Roman Khimov
3646270af0
Merge pull request #2026 from nspcc-dev/fix-state
Fix manifest `Extra` field marshaling
2021-06-29 15:11:25 +03:00
Evgeniy Stratonikov
9c06ba6b07 smartcontract: marshal manifest Extra field as object, fix #2021
This is needed to strip whitespace which can be present.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-06-29 11:39:56 +03:00
Evgeniy Stratonikov
8d67a03aec stackitem: fix Buffer deserialization
Fix incorrect application log for transaction in N3 testnet
8eb4076f1f1c07e693eda7e810779488a2d2b50aba9b727fd237cbc3adbec9e9

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-06-29 11:39:55 +03:00
Evgeniy Stratonikov
f0100407ee compiler/test: compile whole directory in examples
For `nft-nd-nns` example only `namestate.go` file was compiled which is
certainly not what we want.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-06-28 11:50:40 +03:00
Evgeniy Stratonikov
aa76383fa7 compiler: extend permission check to runtime hashes
If a method is known at compile time we can still check
if it is present in the list of methods of at least one contract.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-06-28 11:49:30 +03:00
Evgeniy Stratonikov
4249674ddc compiler: check for contract permissions
On many occassions we can determine at compile-time if contract config lacks
some properties it needs. This includes all native contract invocations
through stdlib, as both hashes and methods are known at compile-time
there.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-06-28 11:49:29 +03:00
Roman Khimov
fecc7a3dd6 keys: simplify code a bit
No functional changes.
2021-06-17 21:55:00 +03:00
Anna Shaleva
d49294b046 core: fix the way candidates are sorted
(*NEO).GetCandidates along with getCandidatesCall use candidates sorted
by serialized ECPoint bytes.

(*NEO).computeCommitteeMembers use candidates sorted by votes, and then
by deserialized ECPoint, i.e. using default ECPoint comparator.
2021-06-16 17:35:04 +03:00
Anna Shaleva
472a44a464 *: regenerate test wallets with resources-friendly scrypt parameters 2021-06-11 15:22:32 +03:00
Anna Shaleva
50296975e2 crypto: allow to provide non-default scrypt parameters 2021-06-11 15:22:28 +03:00
Roman Khimov
61dfca346c
Merge pull request #2003 from carpawell/update-neofs-api-go
Update neofs-api-go
2021-06-11 14:19:12 +03:00
Anna Shaleva
6ff64153db core: check stored native states against autogenerated ones on start 2021-06-10 22:16:05 +03:00
Pavel Karpy
874c56c77b oracle/neofs: stop using deprecated types
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-10 14:16:07 +03:00
Pavel Karpy
c4f8c7a9b2 oracle/neofs: update neofs-api-go
Update neofs-api-go to `v1.27.1`.
Add `WithURIAddress` client option.
Stop using `V2` abstractions outside
of the `api-go` library.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-10 14:16:02 +03:00
Anna Shaleva
be902afe9e core: do not allow NEP17 roundtrip in case of insufficient funds
NEP17 roundtrip is prohibited if from account doesn't have enough funds.

This commit fixes states diff in block 92057 where account
NfuwpaQ1A2xaeVbxWe8FRtaRgaMa8yF3YM initiates two NEO roundtrips with
amount exceeding the account's balance:

block 92057: value mismatch for key +////xTbYWBH3r5qhRKZAPFPHabKfb2vhQ==: QQMhAkwBIQOZZwEA vs QQMhAkwBIQN/ZwEA
block 92057: value mismatch for key +v///ws=: kqlddcitCg== vs tphddcitCg==
block 92057: value mismatch for key +v///xTbYWBH3r5qhRKZAPFPHabKfb2vhQ==: QQEhBUWyDu0W vs QQEhBWmhDu0W

C#'s applog (contains False and False on stack for both transfers):
```
{
   "id" : 1,
   "jsonrpc" : "2.0",
   "result" : {
      "executions" : [
         {
            "gasconsumed" : "11955500",
            "exception" : null,
            "stack" : [
               {
                  "value" : false,
                  "type" : "Boolean"
               },
               {
                  "value" : false,
                  "type" : "Boolean"
               }
            ],
            "vmstate" : "HALT",
            "trigger" : "Application",
            "notifications" : []
         }
      ],
      "txid" : "0x8e73a7e9a566a514813907272ad65fc965002c3b098eacc5bdda529af19d7688"
   }
}
```

Go's applog (both transfers succeeded and GAS minted):
```
{
   "result" : {
      "executions" : [
         {
            "gasconsumed" : "11955500",
            "trigger" : "Application",
            "stack" : [
               {
                  "type" : "Boolean",
                  "value" : true
               },
               {
                  "type" : "Boolean",
                  "value" : true
               }
            ],
            "vmstate" : "HALT",
            "notifications" : [
               {
                  "eventname" : "Transfer",
                  "contract" : "0xd2a4cff31913016155e38e474a2c06d08be276cf",
                  "state" : {
                     "value" : [
                        {
                           "type" : "Any"
                        },
                        {
                           "value" : "22FgR96+aoUSmQDxTx2myn29r4U=",
                           "type" : "ByteString"
                        },
                        {
                           "value" : "4316",
                           "type" : "Integer"
                        }
                     ],
                     "type" : "Array"
                  }
               },
               {
                  "state" : {
                     "type" : "Array",
                     "value" : [
                        {
                           "value" : "22FgR96+aoUSmQDxTx2myn29r4U=",
                           "type" : "ByteString"
                        },
                        {
                           "type" : "ByteString",
                           "value" : "22FgR96+aoUSmQDxTx2myn29r4U="
                        },
                        {
                           "type" : "Integer",
                           "value" : "1111111111"
                        }
                     ]
                  },
                  "contract" : "0xef4073a0f2b305a38ec4050e4d3d28bc40ea63f5",
                  "eventname" : "Transfer"
               },
               {
                  "contract" : "0xef4073a0f2b305a38ec4050e4d3d28bc40ea63f5",
                  "state" : {
                     "type" : "Array",
                     "value" : [
                        {
                           "value" : "22FgR96+aoUSmQDxTx2myn29r4U=",
                           "type" : "ByteString"
                        },
                        {
                           "type" : "ByteString",
                           "value" : "22FgR96+aoUSmQDxTx2myn29r4U="
                        },
                        {
                           "type" : "Integer",
                           "value" : "1111111"
                        }
                     ]
                  },
                  "eventname" : "Transfer"
               }
            ]
         }
      ],
      "txid" : "0x8e73a7e9a566a514813907272ad65fc965002c3b098eacc5bdda529af19d7688"
   },
   "id" : 1,
   "jsonrpc" : "2.0"
}
```
2021-06-09 13:36:50 +03:00
Anna Shaleva
f95bdd9cb5 vm: serialize + rune as \u002B for byte-contained stackitems
This commit fixes the following Go vs C# state diff:

block 74613: value mismatch for key EwAAAHN0cmVhbXMvDg==: eyJpZCI6MTQsInN0YXJ0IjoxNjIyNTAwMjAwMDAwLCJzdG9wIjoxNjIyNTAyMDAwMDAwLCJkZXBvc2l0IjoxMDAwMDAwMDAsInJlbWFpbmluZyI6MTAwMDAwMDAwLCJzZW5kZXIiOiJmeEY4RDl2ZFU3K0gwcDV3NTlyWllMNytNSlE9IiwicmVjaXBpZW50IjoiSVV6c3pveFV0S1NGVnlZRGczSmdTQTFlbTFNPSJ9 vs eyJpZCI6MTQsInN0YXJ0IjoxNjIyNTAwMjAwMDAwLCJzdG9wIjoxNjIyNTAyMDAwMDAwLCJkZXBvc2l0IjoxMDAwMDAwMDAsInJlbWFpbmluZyI6MTAwMDAwMDAwLCJzZW5kZXIiOiJmeEY4RDl2ZFU3XHUwMDJCSDBwNXc1OXJaWUw3XHUwMDJCTUpRPSIsInJlY2lwaWVudCI6IklVenN6b3hVdEtTRlZ5WURnM0pnU0ExZW0xTT0ifQ==

I.e.:
```
{"id":14,"start":1622500200000,"stop":1622502000000,"deposit":100000000,"remaining":100000000,"sender":"fxF8D9vdU7+H0p5w59rZYL7+MJQ=","recipient":"IUzszoxUtKSFVyYDg3JgSA1em1M="}
```
vs
```
{"id":14,"start":1622500200000,"stop":1622502000000,"deposit":100000000,"remaining":100000000,"sender":"fxF8D9vdU7\u002BH0p5w59rZYL7\u002BMJQ=","recipient":"IUzszoxUtKSFVyYDg3JgSA1em1M="}
```
2021-06-08 16:40:38 +03:00
Anna Shaleva
20e0a0d746 core: add price for CALLT instruction 2021-06-07 19:17:22 +03:00
Roman Khimov
352450d25a
Merge pull request #1988 from nspcc-dev/cli-permission
cli,compiler: allow to specify manifest permissions
2021-06-07 10:27:29 +03:00
Roman Khimov
2837b98eb3
Merge pull request #1993 from nspcc-dev/mpt-tests
Add MPT compatibility tests
2021-06-07 10:24:59 +03:00
Roman Khimov
a6fc5d9fe4
Merge pull request #1998 from nspcc-dev/fix-calculatenetworkfee-result
rpc: wrap calculatenetworkfee result in a structure
2021-06-07 09:53:48 +03:00
Roman Khimov
efb814a97e
Merge pull request #1995 from nspcc-dev/compiler-verify-notify
compiler: disallow `runtime.Notify` in `Verify` function
2021-06-06 11:34:42 +03:00
Evgeniy Stratonikov
1578904da2 compiler: disallow runtime.Notify in Verify function
Only direct invocations are considered. The check can be disabled
with '--no-events' compiler option.
2021-06-06 09:34:24 +03:00
Roman Khimov
1cea0dd894
Merge pull request #1997 from nspcc-dev/drop-syncreached-check
network: drop useless flag check
2021-06-04 23:39:34 +03:00
Roman Khimov
6a61e86d12 rpc: wrap calculatenetworkfee result in a structure
C#:
{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "networkfee": "1185120"
  }
}

Go:
{
  "id": 2,
  "jsonrpc": "2.0",
  "result": 1185120
}

Thanks @csmuller for finding it.
2021-06-04 23:38:35 +03:00
Roman Khimov
f6da88af0d network: drop useless flag check
It's the first thing done in tryStartServices(), so checking it here doesn't
make much sense.
2021-06-04 20:29:47 +03:00
Evgeniy Stratonikov
4467e2cbfd cli,compiler: allow to specify manifest permissions 2021-06-04 11:16:22 +03:00
Evgeniy Stratonikov
f3d02b9fef mpt/tests: add compatibility tests 2021-06-04 11:09:17 +03:00
Evgeniy Stratonikov
e833d333fe mpt: disallow empty keys
This is not a problem in practice, as all keys are prefixed
by a contract ID. However in theory it can lead to a different
state root after new portion of changes thus this fix.
2021-06-04 11:09:17 +03:00
Evgeniy Stratonikov
5b1f6207de mpt: reduce max key length
The constant used is actually a nibbled-path restriction.
Clarify this and make them private.
2021-06-04 11:09:17 +03:00
Evgeniy Stratonikov
1d6d7206e9 compiler: inline expressions with type conversions, fix #1879
Don't count `string(data)` or `[]byte(data)` as function calls.
2021-06-04 11:08:00 +03:00
Evgeniy Stratonikov
6b3afe9131 compiler: inline all expression with no side-effects, fix #1881
This has the drawback of traversing expression each time the argument is
used. However this is not the case in our syscall/native wrappers.
The old behaviour can be restored by explicit local assignment.
2021-06-04 11:07:59 +03:00
Evgeniy Stratonikov
3749f38720 compiler/test: add more tests for inline behaviour 2021-06-04 11:06:09 +03:00
Evgeniy Stratonikov
c538512e60 compiler/test: check Main locals in inline tests 2021-06-04 11:06:09 +03:00
Evgeniy Stratonikov
5e92a254b2 compiler: check event name length in runtime.Notify 2021-06-03 10:32:56 +03:00
Evgeniy Stratonikov
1a5e656d38 compiler: restore event check warnings
It was broken after changes for inlining. Add tests.
2021-06-03 10:32:56 +03:00
Anna Shaleva
b14fc5da7c vm: increase waiting time for vm cli program 2021-06-01 16:29:08 +03:00
Anna Shaleva
1dbf1d4310 rpc: allow to track notary requests via Notification subsystem 2021-06-01 16:29:04 +03:00
Anna Shaleva
b8e96ac82b core: move mempool.Event to a separate package
And write a marshaller for EventType, it'll be reused by the Notification
subsystem.
2021-06-01 12:24:28 +03:00
Anna Shaleva
999fba81cd core: export (*NEOBalanceState).FromStackItem 2021-05-31 11:18:54 +03:00
Roman Khimov
b858be63b4
Merge pull request #1969 from nspcc-dev/examples/nns-fix
examples: fix IPv6 bounds check
2021-05-28 14:29:54 +03:00
Anna Shaleva
e3611bfa4c vm, cli: allow to specify flags while loading VM 2021-05-28 12:07:41 +03:00
Anna Shaleva
2a9ce03376 examples: fix IPv6 bounds check 2021-05-28 11:31:09 +03:00
Evgeniy Stratonikov
233996eec0 native/neo: add getAccountState method, close #1975 2021-05-27 10:59:38 +03:00
Evgeniy Stratonikov
e314e55a1b compiler: add Base58Check* interop wrappers
Missing piece from #1977.
2021-05-25 18:06:40 +03:00