Commit graph

4897 commits

Author SHA1 Message Date
Anna Shaleva
6d3cc0f7e7 core: extend error text for storage context in dynamic script 2022-11-28 17:17:36 +03:00
Anna Shaleva
0ceae612c6 vm: improve failed SYSCALL logging
For better user experience.
2022-11-28 17:16:27 +03:00
Roman Khimov
07e32e221d *: update interops, add System.Runtime.LoadScript wrapper check 2022-11-25 15:16:27 +03:00
Roman Khimov
de2de986a7 interop: add runtime.LoadScript 2022-11-25 15:11:49 +03:00
Roman Khimov
4e26f4b26e runtime: implement System.Runtime.LoadScript, fix #2701 2022-11-25 15:11:49 +03:00
Roman Khimov
1c38b45074 core: don't always store all hashes in memory
We're paging these hashes, so we need a previous full page and a current one
plus some cache for various requests. Storing 1M of hashes is 32M of memory
and it grows quickly. It also seriously affects node startup time, most of
what it's doing is reading these hashes, the longer the chain the more time it
needs to do that.

Notice that this doesn't change the underlying DB scheme in any way.
2022-11-25 14:30:51 +03:00
Roman Khimov
0ad6e295ea core: make GetHeaderHash accept uint32
It should've always been this way because block indexes are uint32.
2022-11-25 14:30:51 +03:00
Roman Khimov
16db4746cc core: drop another useless genesis-specific header handling branch
If we only have genesis block (or <2000 headers) then we might as well use
generic logic below with zero targetHash because genesis block has zero
PrevHash (and its hash will naturally be the last on the chain going
backwards).
2022-11-25 13:08:03 +03:00
Roman Khimov
83e2ca054f core: simplify header hash list restoration logic
We don't care about header, we only need hashes here.
2022-11-25 13:08:03 +03:00
Roman Khimov
ff284d5d5c core: remove genesis-specific header hash logic from init
The code below will do the same thing (via CreateGenesisBlock, but it doesn't
matter) generically, so this branch is useless.
2022-11-25 13:08:03 +03:00
Roman Khimov
027e94fbde compiler: don't push overrides for unexported methods
They're internal, no one cares about them externally.
2022-11-23 17:05:37 +03:00
Roman Khimov
327dfb8dbf rpcsrv: add wsclient error message test for #2818
Now it can be done easily.
2022-11-23 12:24:20 +03:00
Roman Khimov
2591c39500 rpcsrv: make websocket client limit configurable 2022-11-23 12:19:49 +03:00
Roman Khimov
fb09670fd1 rpcclient: extract more detailed server-side on WS connection problem
If available. Fixes #2818.
2022-11-23 12:19:35 +03:00
Roman Khimov
ab0b23625b
Merge pull request #2817 from nspcc-dev/always-return-hash-vub-from-sender
Some actor/waiter interaction fixes
2022-11-23 14:37:38 +07:00
Roman Khimov
fd04b2befd actor: don't abort waiter on "already exists" error
It can happen in many cases of distributed tx generation/submission, we can
just wait normally in this case and there will be some proper result.
2022-11-23 10:07:37 +03:00
Roman Khimov
66ddeccdad
Merge pull request #2813 from nspcc-dev/fix-state-reset
core: fix broken state reset
2022-11-23 13:43:42 +07:00
Anna Shaleva
f3ef2890f0 core: check headers at the proper state on state reset
And fix the comment along the way.
2022-11-23 09:16:33 +03:00
Roman Khimov
cd6bb68246 actor: check for tx after the subscription in wsWaiter, fix #2805
Don't wait for VUB block, solve this race immediately.
2022-11-22 17:28:55 +03:00
Roman Khimov
c95d140113 rpcclient: always return tx hash from sendrawtransaction
Let upper-layer APIs like actor.Send() return it as well. Server can return
"already exists" which is an error and yet at the same time a very special
one, in many cases it means we can proceed with waiting for the TX to settle.
2022-11-22 15:18:37 +03:00
Anna Shaleva
b27a9bcf95 core: adjust info message for proper-stated chains
Make it prettier for those cases when `db reset` command was called
after interrupted reset.
2022-11-22 11:53:39 +03:00
Anna Shaleva
b82374823e core: increase persist batch size for reset storage changes 2022-11-22 11:53:39 +03:00
Anna Shaleva
bdc42cd595 core: reset blocks, txs and AERs in several stages
Sometimes it can be hard to persist all changes at ones, the process
can take almost all RAM and a lot of time. Here's the example of reset
for mainnet from 2.4M to 1:
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -m --height 1
2022-11-20T17:16:48.236+0300	INFO	MaxBlockSize is not set or wrong, setting default value	{"MaxBlockSize": 262144}
2022-11-20T17:16:48.236+0300	INFO	MaxBlockSystemFee is not set or wrong, setting default value	{"MaxBlockSystemFee": 900000000000}
2022-11-20T17:16:48.237+0300	INFO	MaxTransactionsPerBlock is not set or wrong, using default value	{"MaxTransactionsPerBlock": 512}
2022-11-20T17:16:48.237+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T17:16:48.240+0300	INFO	restoring blockchain	{"version": "0.2.6"}
2022-11-20T17:16:48.297+0300	INFO	initialize state reset	{"target height": 1}
2022-11-20T17:16:48.300+0300	INFO	trying to reset blocks, transactions and AERs
2022-11-20T17:19:29.313+0300	INFO	blocks, transactions ans AERs are reset	{"took": "2m41.015126493s", "keys": 3958420}
...
```
To avoid OOM killer, split blocks reset into multiple stages. It increases
operation time due to intermediate DB persists, but makes things cleaner, the
result for almost the same DB height with the new approach:
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -m --height 1
2022-11-20T17:39:42.023+0300	INFO	MaxBlockSize is not set or wrong, setting default value	{"MaxBlockSize": 262144}
2022-11-20T17:39:42.023+0300	INFO	MaxBlockSystemFee is not set or wrong, setting default value	{"MaxBlockSystemFee": 900000000000}
2022-11-20T17:39:42.023+0300	INFO	MaxTransactionsPerBlock is not set or wrong, using default value	{"MaxTransactionsPerBlock": 512}
2022-11-20T17:39:42.023+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T17:39:42.026+0300	INFO	restoring blockchain	{"version": "0.2.6"}
2022-11-20T17:39:42.071+0300	INFO	initialize state reset	{"target height": 1}
2022-11-20T17:39:42.073+0300	INFO	trying to reset blocks, transactions and AERs
2022-11-20T17:40:11.735+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 1, "took": "29.66363737s", "keys": 210973}
2022-11-20T17:40:33.574+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 2, "took": "21.839208683s", "keys": 241203}
2022-11-20T17:41:29.325+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 3, "took": "55.750698386s", "keys": 250593}
2022-11-20T17:42:12.532+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 4, "took": "43.205892757s", "keys": 321896}
2022-11-20T17:43:07.978+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 5, "took": "55.445398156s", "keys": 334822}
2022-11-20T17:43:35.603+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 6, "took": "27.625292032s", "keys": 317131}
2022-11-20T17:43:51.747+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 7, "took": "16.144359017s", "keys": 355832}
2022-11-20T17:44:05.176+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 8, "took": "13.428733899s", "keys": 357690}
2022-11-20T17:44:32.895+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 9, "took": "27.718548783s", "keys": 393356}
2022-11-20T17:44:51.814+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 10, "took": "18.917954658s", "keys": 366492}
2022-11-20T17:45:07.208+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 11, "took": "15.392642196s", "keys": 326030}
2022-11-20T17:45:18.776+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 12, "took": "11.568255716s", "keys": 299884}
2022-11-20T17:45:25.862+0300	INFO	last batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 13, "took": "7.086079594s", "keys": 190399}
2022-11-20T17:45:25.862+0300	INFO	blocks, transactions ans AERs are reset	{"took": "5m43.791214084s", "overall persisted keys": 3966301}
...
```
2022-11-22 11:53:39 +03:00
Anna Shaleva
d67f0df516 core: reset block headers together with header height info
We need to keep the headers information consistent with header batches
and headers. This comit fixes the bug with failing blockchain
initialization on recovering from state reset interrupted after the
second stage (blocks/txs/AERs removal):
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -t --height 83000
2022-11-20T16:28:29.437+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T16:28:29.440+0300	INFO	restoring blockchain	{"version": "0.2.6"}
failed to create Blockchain instance: could not initialize blockchain: could not get header 1898cd356a4a2688ed1c6c7ba1fd6ba7d516959d8add3f8dd26232474d4539bd: key not found
```
2022-11-22 11:53:39 +03:00
Anna Shaleva
283da8f599 core: use DAO-provided block height during during state reset
Don't use cache because it's not yet initialized. Also, perform
safety checks only if state reset wasn't yet started. These fixes
alloww to solve the following problem while recovering from
interrupted state reset:
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -t --height 83000
2022-11-20T15:51:31.431+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T15:51:31.434+0300	INFO	restoring blockchain	{"version": "0.2.6"}
failed to create Blockchain instance: could not initialize blockchain: current block height is 0, can't reset state to height 83000
```
2022-11-22 11:53:39 +03:00
Anna Shaleva
7d55bf2cc1 core: log persisted storage item batches count during state reset 2022-11-22 11:53:39 +03:00
Anna Shaleva
f52451e582 core: fix state reset with broken contract
Sync up with #2802, bad contract -> no contract ID at all.
2022-11-22 11:53:39 +03:00
Anna Shaleva
ecda07736e core: stop storage items reset after any seek error 2022-11-22 11:53:39 +03:00
Anna Shaleva
bfe7aeae7b core: stop storage items reset after the first persist error
It's a bug, we mustn't continue if something bad had happend on persist,
otherwise this error will be overwritten by subsequent successfull persist.
2022-11-22 11:53:39 +03:00
Anna Shaleva
235518eb6c core: reset batch counter to zero after each persist in resetStateInternal
It's a bug, otherwise we'll persist each storage item after 10K-th one,
that's the reason of abnormous long storage items resetting stage.
2022-11-22 11:53:39 +03:00
Anna Shaleva
9f23fafc03 core: improve logging of resetStateInternal
Inform when starting subsequent stage, inform about keys persisted.
2022-11-22 11:53:36 +03:00
Roman Khimov
0039615ae3
Merge pull request #2816 from nspcc-dev/fix-pointer-serialization
Fix pointer serialization
2022-11-20 22:59:52 +07:00
Roman Khimov
9ba18b5dfa stackitem: serialize/deserialize pointers, fix #2815
They of course can't be serialized, but in protected mode we still need to
handle them somehow.
2022-11-20 16:02:50 +03:00
Roman Khimov
48140320db
Merge pull request #2812 from nspcc-dev/improve-vm-context-handling
Improve vm istack/estack handling
2022-11-20 19:42:35 +07:00
Roman Khimov
ca9fde745b
Merge pull request #2809 from nspcc-dev/fix-subs
rpcsrv: do not block blockchain events receiver by subscription requests
2022-11-18 16:16:41 +07:00
Roman Khimov
8e7f65be17 vm: use proper estack for exception handler
v.estack might be some inner invoked contract and its stack must not be used
for exception handler set up by higher-order contract.
2022-11-18 11:36:38 +03:00
Roman Khimov
cb64957af5 vm: don't use Stack for istack
We don't use all of the Stack functionality for it, so drop useless methods
and avoid some interface conversions. It increases single-node TPS by about
0.9%, so nothing really important there, but not a bad change either. Maybe it
can be reworked again with generics though.
2022-11-18 11:35:29 +03:00
Anna Shaleva
4df9a5e379 rpcsrv: refactor subscribe routine
Move shutdown check after subsCounterLock is taken in the end of
`(s *Server) subscribe` in order to avoid extra locks holding.
2022-11-18 10:54:10 +03:00
Anna Shaleva
b7f19a54d5 services: fix chain locked by WS subscriptions handlers
Blockchain's subscriptions, unsubscriptions and notifications are
handled by a single notificationDispatcher routine. Thus, on attempt
to send the subsequent event to Blockchain's subscribers, dispatcher
can't handle subscriptions\unsubscriptions. Make subscription and
unsubscription to be a non-blocking operation for blockchain on the
server side, otherwise it may cause the dispatcher locks.

To achieve this, use a separate lock for those code that make calls
to blockchain's subscription API and for subscription counters on
the server side.
2022-11-18 09:30:12 +03:00
Roman Khimov
2bcb7bd06f compiler: don't use (*VM).Istack when it's not needed 2022-11-17 20:46:06 +03:00
Roman Khimov
b8c09f509f network: add random slight delay to connection attempts
Small (especially dockerized/virtualized) networks often start all nodes at
ones and then we see a lot of connection flapping in the log. This happens
because nodes try to connect to each other simultaneously, establish two
connections, then each one finds a duplicate and drops it, but this can be
different duplicate connections on other sides, so they retry and it all
happens for some time. Eventually everything settles, but we have a lot of
garbage in the log and a lot of useless attempts.

This random waiting timeout doesn't change the logic much, adds a minimal
delay, but increases chances for both nodes to establish a proper single
connection on both sides to only then see another one and drop it on both
sides as well. It leads to almost no flapping in small networks, doesn't
affect much bigger ones. The delay is close to unnoticeable especially if
there is something in the DB for node to process during startup.
2022-11-17 18:42:43 +03:00
Roman Khimov
075a54192c network: don't try too many connections
Consider mainnet, it has an AttemptConnPeers of 20, so may already have 3
peers and request 20 more, then have 4th connected and attemtp 20 more again,
this leads to a huge number of connections easily.
2022-11-17 18:03:04 +03:00
Roman Khimov
6bce973ac2 network: drop duplicationg check from handleAddrCmd()
It was relevant with the queue-based discoverer, now it's not, discoverer
handles this internally.
2022-11-17 17:42:36 +03:00
Roman Khimov
1c7487b8e4 network: add a timer to check for peers
Consider initial connection phase for public networks:
 * simultaneous connections to seeds
 * very quick handshakes
 * got five handshaked peers and some getaddr requests sent
 * but addr replies won't trigger new connections
 * so we can stay with just five connections until any of them breaks or a
   (long) address checking timer fires

This new timers solves the problem, it's adaptive at the same time. If we have
enough peers we won't be waking up often.
2022-11-17 17:32:05 +03:00
Anna Shaleva
e73c3c7ec4 services: adjust WS waiter test
Make it more stable.
2022-11-17 17:15:01 +03:00
Roman Khimov
23f118a1a9 network: rework discoverer/server interaction
* treat connected/handshaked peers separately in the discoverer, save
   "original" address for connected ones, it can be a name instead of IP and
   it's important to keep it to avoid reconnections
 * store name->IP mapping for seeds if and when they're connected to avoid
   reconnections
 * block seed if it's detected to be our own node (which is often the case for
   small private networks)
 * add an event for handshaked peers in the server, connected but
   non-handshaked ones are not really helpful for MinPeers or GetAddr logic

Fixes #2796.
2022-11-17 17:07:19 +03:00
Roman Khimov
6ba4afc977 network: consider handshaked peers only when comparing with MinPeers
We don't know a lot about non-handshaked ones, so it's safer to try more
connection.
2022-11-17 16:40:29 +03:00
Roman Khimov
ab0ff63ce1
Merge pull request #2804 from nspcc-dev/check-aer-sub
rpc: fix subscribers locking logic and properly drain poll-based waiter receiver
2022-11-17 04:24:35 +07:00
Anna Shaleva
1399496dfb rpcclient: refactor event-based waiting loop
Avoid receiver channels locks.
2022-11-16 23:57:00 +03:00
Anna Shaleva
95e23c8e46 actor: fix event-based tx awaiting
If VUB-th block is received, we still can't guaranty that transaction
wasn't accepted to chain. Back this situation by rolling back to a
poll-based waiter.
2022-11-16 23:44:31 +03:00
Anna Shaleva
6dbae7edc4 rpcclient: fix WS-client unsubscription process
Do not block subscribers until the unsubscription request to RPC server
is completed. Otherwise, another notification may be received from the
RPC server which will block the unsubscription process.

At the same time, fix event-based waiter. We must not block the receiver
channel during unsubscription because there's a chance that subsequent
event will be sent by the server. We need to read this event in order not
to block the WSClient's readloop.
2022-11-16 23:44:30 +03:00
Anna Shaleva
ddaba9e74d rpcsrv: fix "subscribe" parameters handling
If it's a subscription for AERs, we need to check the filter's state only
if it has been provided, otherwise filter is always valid.
2022-11-16 14:05:13 +03:00
Anna Shaleva
d043139b66 rpcsrv: adjust "subscribe" response error
Make it more detailed for better debugging experience.
2022-11-16 13:35:19 +03:00
Anna Shaleva
3f122fd591 rpcclient: adjust WS waiter error formatting
Follow the other errors formatting style.
2022-11-16 12:22:18 +03:00
Roman Khimov
822722bd2e native: ignore decoding errors during cache init
Bad contract -> no contract. Unfortunately we've got a broken
6f1837723768f27a6f6a14452977e3e0e264f2cc contract on the mainnet which can't
be decoded (even though it had been saved successfully), so this is a
temporary fix for #2801 to be able to start mainnet node after shutdown.
2022-11-16 12:00:28 +03:00
Roman Khimov
aef01bf663 vm: fix istack marshaling, fix #2799 2022-11-16 00:40:12 +03:00
Roman Khimov
90582faacd vm: save current stack slice when loading new context
v.estack is used throughout the code to work with estack, while ctx.sc.estack
is (theoretically) just a reference to it that is saved on script load and
restored to v.estack on context unload. The problem is that v.estack can grow
as we use it and can be reallocated away from its original slice (saved in the
ctx.sc.estack), so either ctx.sc.estack should be a pointer or we need to
ensure that it's correct when loading a new script. The second approach is a
bit safer for now and it fixes #2798.
2022-11-15 23:48:02 +03:00
Anna Shaleva
6f3a0a6b4c network: adjust warning for deposit expiration
Provide additional info for better user experience.
2022-11-15 14:16:34 +03:00
Roman Khimov
c67ee54566
Merge pull request #2792 from nspcc-dev/rpcwrapper-arrays
RPC wrapper for simple arrays
2022-11-15 13:08:25 +07:00
Roman Khimov
82c6ce218b rpcbinding: use binding condig to generate code for simple arrays
Part of #2767.
2022-11-14 13:01:13 +03:00
Roman Khimov
b5c79f4be3 unwrap: add a complete set of simple array unwrappers
Arrays of basic types should be covered completely.
2022-11-14 13:01:13 +03:00
Roman Khimov
c405092953 network: pre-filter transactions going into dbft
Drop some load from dbft loop during consensus process.
2022-11-11 15:32:51 +03:00
Roman Khimov
f78231fd9c
Merge pull request #2773 from nspcc-dev/state-reset
core: implement state reset
2022-11-10 22:26:43 +07:00
Anna Shaleva
e7effef052 core: reuse appendTokenTransferInfo from appendTokenTransfer 2022-11-10 18:08:18 +03:00
Anna Shaleva
bd6bb9e9e2 core: allow to reset blockchain state 2022-11-10 18:08:17 +03:00
Anna Shaleva
ec9317d5b4 core: fix typo 2022-11-10 17:58:06 +03:00
Roman Khimov
cb806d4233 result: drop Version (un)marshaler, we only need it for Protocol
Simplify code a bit.
2022-11-10 17:51:02 +03:00
Roman Khimov
8324a247d3 result: drop pre-0.99.0 compatibility code
0.99.0 is too old already.
2022-11-10 16:49:38 +03:00
Anna Shaleva
1dac45bbbb core: add ability to check whether blockchain is running 2022-11-10 16:47:04 +03:00
Roman Khimov
7f8a79ffaa result: drop deprecated Magic and StateRootInHeader from Version
It's more than a year now we have them deprecated.
2022-11-10 16:32:49 +03:00
Roman Khimov
4c9473872e
Merge pull request #2783 from nspcc-dev/rpcbindings-iterators
Initial iterator support for RPC bindings
2022-11-10 12:51:40 +07:00
Roman Khimov
145ebad90e binding: drop the only error condition from TemplateFromManifest
Simplify the interface, we do IsValid() check anyway in the CLI and it covers
this condition as well.
2022-11-09 18:13:45 +03:00
Roman Khimov
be02eea7b1 binding: precompile template, remove useless error condition 2022-11-09 18:13:45 +03:00
Roman Khimov
a7f86dcb7f rpcbinding: generate Expanded methods for iterators
Refs. #2768.
2022-11-09 18:13:45 +03:00
Roman Khimov
d569fe01e6 rpcbinding: initial support for iterators, see #2768
Already better than stackitem.Item.
2022-11-09 18:13:45 +03:00
Roman Khimov
d0b1015b2c rpcsrv: provide Prometheus histograms for calls
They're a bit more useful and they're naturally grouped under rpc
prefix. Simple counters will be removed eventually to avoid duplication.
2022-11-09 13:26:45 +03:00
Roman Khimov
3247aa40a7 rpcsrv: allow any Origin in WS connections if EnableCORSWorkaround
Break origin checks even more. Alternative to #2772.
2022-11-09 09:28:09 +03:00
Roman Khimov
e3c503cee9
Merge pull request #2781 from nspcc-dev/btcd-update
Update 256k1 dependency
2022-11-08 23:50:19 +07:00
Roman Khimov
1e07cacc79 rpcsrv: fix test
client_test.go:1935:
                Error Trace:    /home/rik/dev/neo-go/pkg/services/rpcsrv/client_test.go:1935
                Error:          Should NOT be empty, but was 00000000-0000-0000-0000-000000000000
                Test:           TestClient_Iterator_SessionConfigVariations/sessions_disabled

It's obviously empty, since we have sessions disabled, but it was not
considered to be empty in testify 1.7.0, now it is, see 840cb80149
2022-11-08 18:31:25 +03:00
Roman Khimov
8746d9877e keys: move from btcd/btcec to decred/secp256k1 package
There is a security issue found in github.com/btcsuite/btcd that we don't care
about (we're only using 256k1 implementation), but GitHub complains about
it. We could update to github.com/btcsuite/btcd/btcec/v2, but it's now just a
thin wrapper over github.com/decred/dcrd/dcrec/secp256k1/v4, so we better use
it directly.
2022-11-08 17:59:59 +03:00
Roman Khimov
69d8905ad9 rpcbinding: exclude onNEPXXPayment methods from wrappers
They make no sense there.
2022-11-08 17:01:36 +03:00
Roman Khimov
130608ac67 rpcbinding: support writer-only wrappers
"verify" contract doesn't have any safe methods.
2022-11-08 17:01:36 +03:00
Roman Khimov
df29008a50 rpcbinding: add GAS testcase, fix methodless wrappers
* strip NEP-XX methods before going into generator to avoid unused imports
 * nepXX.Invoker types already include Call
 * always import util, it's used for Hash
2022-11-08 17:01:36 +03:00
Roman Khimov
aeb61fb61d rpcbinding: generate ASSERT for bool-returning methods
It's a common pattern.
2022-11-08 17:01:36 +03:00
Roman Khimov
2a4a5ab479 rpcbinding: support simple wrappers for writer methods
Fixes #2769.
2022-11-08 17:01:36 +03:00
Roman Khimov
bb47d971dc rpcclient/nep11: make DivisibleWriter part reusable
Same rationale as for BaseWriter or nep17.TokenWriter, otherwise it's hard to
build on top of plain Divisible structure.
2022-11-07 10:44:39 +03:00
Roman Khimov
b590d4ca04
Merge pull request #2766 from nspcc-dev/rpc-wrapper-autogen
Rpc wrapper autogeneration
2022-10-28 11:54:47 +07:00
Roman Khimov
02ce59cfd5 binding: avoid name conflicts with Go keywords
And clashing one name on another after rename.
2022-10-27 22:57:49 +03:00
Roman Khimov
617c31093f smartcontract: initial rpcbinding implementation, fix #2705
It can do some unwrapping and reuse nepXX packages. It only uses manifest data
at the moment, see #2767, #2768, #2769.
2022-10-27 22:57:49 +03:00
Roman Khimov
f0abc035af unwrap: add PublicKey to unwrap public keys
We have this type in NEP-14 directly.
2022-10-27 18:32:00 +03:00
Roman Khimov
e0eff94094 standard: correct Comply* comments 2022-10-27 18:32:00 +03:00
Roman Khimov
64b603b056 standard: export standard definitions
Make Comply* functions useful and expose standard definitions for some reuse.
2022-10-27 18:32:00 +03:00
Anna Shaleva
4fc11c2924 rpc: revert deprecated subscriptions API changes
Revert deprecated subscriptions-related method signature changed in
673a495527, 8e84bb51d5
and d7c1f3eac7.
2022-10-26 15:32:54 +03:00
Anna Shaleva
5811687836 rpc: fix bug in Actor's waiter
Execution events are followed by block events, not vise versa, thus,
we can wait until VUB block to be accepted to be sure that
transaction wasn't accepted to chain.
2022-10-26 15:32:54 +03:00
Anna Shaleva
0a5905390c rpc: refactor WSClient subscriptions API
Make it more specific, close #2756.
2022-10-26 15:32:54 +03:00
Anna Shaleva
2a53db42af neorpc: adjust and extend event filters documentation 2022-10-26 15:32:54 +03:00
Roman Khimov
396f56f792 binding: drop unused lowerFirst 2022-10-26 12:47:39 +03:00
Roman Khimov
4191b18728 binding: make TemplateFromManifest more reusable
Other template generators can make use of it.
2022-10-26 12:43:48 +03:00
Roman Khimov
d6d4f07280 binding: always import some packages and do it outside
If the contract has no methods, it's probably a broken one.
2022-10-26 12:40:59 +03:00
Roman Khimov
6af71755c1
Merge pull request #2762 from nspcc-dev/optimize-emit-bool
emit: optimize Bool GAS cost
2022-10-26 15:06:05 +07:00
Roman Khimov
4e58bd7411 compiler: use shorter and cheaper sequence to convert to Boolean 2022-10-25 18:20:55 +03:00
Roman Khimov
5d43367082 emit: optimize Bool GAS cost
NOT is 1 byte shorter and 2048 times cheaper than CONVERT. Inspired by
neo-project/neo-vm#493.
2022-10-25 13:08:33 +03:00
Roman Khimov
e19d867d4e
Merge pull request #2761 from nspcc-dev/fancy-getaddr
Fancy getaddr
2022-10-25 16:51:38 +07:00
Roman Khimov
28f54d352a network: do getaddr requests periodically, fix #2745
Every 1000 blocks seems to be OK for big networks (that only had done some
initial requests previously and then effectively never requested addresses
again because there was a sufficient number of addresses), won't hurt smaller
ones as well (that effectively keep doing this on every connect/disconnect,
peer changes are very rare there, but when they happen we want to have some
quick reaction to these changes).
2022-10-24 15:10:51 +03:00
Roman Khimov
9efc110058 network: it is 42
32 is a very good number, but we all know 42 is a better one. And it can even
be proven by tests with higher peaking TPS values.

You may wonder why is it so good? Because we're using packet-switching
networks mostly and a packet is a packet almost irrespectively of how bit it
is. Yet a packet has some maximum possible size (hi, MTU) and this size most
of the time is 1500 (or a little less than that, hi VPN). Subtract IP header
(20 for IPv4 or 40 for IPv6 not counting options), TCP header (another 20) and
Neo message/payload headers (~8 for this case) and we have just a little more
than 1400 bytes for our dear hashes. Which means that in a single packet most
of the time we can have 42-44 of them, maybe 45. Choosing between these
numbers is not hard then.
2022-10-24 14:44:19 +03:00
Roman Khimov
9d6b18adec network: drop minPoolCount magic constant
We have AttemptConnPeers that is closely related, the more we have there the
bigger the network supposedly is, so it's much better than magic minPoolCount.
2022-10-24 14:36:10 +03:00
Roman Khimov
af24051bf5 network: sleep a bit before retrying reconnects
If Dial() is to exit quickly we can end up in a retry loop eating CPU.
2022-10-24 14:34:48 +03:00
Roman Khimov
a17d9f80a4
Merge pull request #2759 from nspcc-dev/avoid-db-lookup-in-hasblock
core: optimize HasBlock check for recent blocks
2022-10-24 14:33:38 +07:00
Roman Khimov
f42b8e78fc
Merge pull request #2758 from nspcc-dev/check-inflight-tx-invs
network: check inv against currently processed transactions
2022-10-24 14:16:33 +07:00
Roman Khimov
52df03ecbc
Merge pull request #2757 from nspcc-dev/batched-getdata
network: batch getdata replies
2022-10-24 14:15:55 +07:00
Roman Khimov
0551ddff4e
Merge pull request #2749 from nspcc-dev/rpc-waiter
rpc: implement transaction awaiting
2022-10-24 13:08:06 +07:00
Anna Shaleva
3cccc89dac rpc: add Wait wrapper to Notary actor 2022-10-24 06:09:36 +03:00
Anna Shaleva
1a6f1c805c rpc: fix race in TestWSClientEvents
```
2022-10-21T08:59:45.2219797Z === RUN   TestWSClientEvents/default_ntf_channel
2022-10-21T08:59:45.2219901Z ==================
2022-10-21T08:59:45.2220017Z WARNING: DATA RACE
2022-10-21T08:59:45.2220177Z Write at 0x00c000c82778 by goroutine 371:
2022-10-21T08:59:45.2220580Z   github.com/nspcc-dev/neo-go/pkg/rpcclient.TestWSClientEvents.func2()
2022-10-21T08:59:45.2221112Z       /home/runner/work/neo-go/neo-go/pkg/rpcclient/wsclient_test.go:171 +0x1c4
2022-10-21T08:59:45.2221244Z   testing.tRunner()
2022-10-21T08:59:45.2221617Z       /opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1439 +0x213
2022-10-21T08:59:45.2221759Z   testing.(*T).Run.func1()
2022-10-21T08:59:45.2222124Z       /opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1486 +0x47
2022-10-21T08:59:45.2222138Z
2022-10-21T08:59:45.2222308Z Previous read at 0x00c000c82778 by goroutine 37:
2022-10-21T08:59:45.2222694Z   github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).StateRootInHeader()
2022-10-21T08:59:45.2223151Z       /home/runner/work/neo-go/neo-go/pkg/rpcclient/rpc.go:1104 +0xb0
2022-10-21T08:59:45.2223482Z   github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).wsReader()
2022-10-21T08:59:45.2224077Z       /home/runner/work/neo-go/neo-go/pkg/rpcclient/wsclient.go:210 +0x651
2022-10-21T08:59:45.2224416Z   github.com/nspcc-dev/neo-go/pkg/rpcclient.NewWS.func2()
2022-10-21T08:59:45.2224892Z       /home/runner/work/neo-go/neo-go/pkg/rpcclient/wsclient.go:149 +0x39
2022-10-21T08:59:45.2224901Z
2022-10-21T08:59:45.2225049Z Goroutine 371 (running) created at:
2022-10-21T08:59:45.2225182Z   testing.(*T).Run()
2022-10-21T08:59:45.2225548Z       /opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1486 +0x724
2022-10-21T08:59:45.2225911Z   github.com/nspcc-dev/neo-go/pkg/rpcclient.TestWSClientEvents()
2022-10-21T08:59:45.2226408Z       /home/runner/work/neo-go/neo-go/pkg/rpcclient/wsclient_test.go:167 +0x404
2022-10-21T08:59:45.2226539Z   testing.tRunner()
2022-10-21T08:59:45.2226900Z       /opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1439 +0x213
2022-10-21T08:59:45.2227042Z   testing.(*T).Run.func1()
2022-10-21T08:59:45.2227398Z       /opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1486 +0x47
2022-10-21T08:59:45.2227406Z
2022-10-21T08:59:45.2227552Z Goroutine 37 (finished) created at:
2022-10-21T08:59:45.2227851Z   github.com/nspcc-dev/neo-go/pkg/rpcclient.NewWS()
2022-10-21T08:59:45.2228327Z       /home/runner/work/neo-go/neo-go/pkg/rpcclient/wsclient.go:149 +0x6fb
2022-10-21T08:59:45.2228843Z   github.com/nspcc-dev/neo-go/pkg/rpcclient.TestWSClientEvents.func2()
2022-10-21T08:59:45.2229434Z       /home/runner/work/neo-go/neo-go/pkg/rpcclient/wsclient_test.go:168 +0x131
2022-10-21T08:59:45.2229569Z   testing.tRunner()
2022-10-21T08:59:45.2229930Z       /opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1439 +0x213
2022-10-21T08:59:45.2230069Z   testing.(*T).Run.func1()
2022-10-21T08:59:45.2230424Z       /opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1486 +0x47
2022-10-21T08:59:45.2230526Z ==================
2022-10-21T08:59:45.2230703Z     wsclient_test.go:186:
2022-10-21T08:59:45.2230988Z         	Error Trace:	wsclient_test.go:186
2022-10-21T08:59:45.2231209Z         	Error:      	Should be true
2022-10-21T08:59:45.2231536Z         	Test:       	TestWSClientEvents/default_ntf_channel
2022-10-21T08:59:45.2231812Z     testing.go:1312: race detected during execution of test
```
2022-10-24 06:09:36 +03:00
Anna Shaleva
d2a9e9120d rpc: extend Waiter interface to wait for several txs with context 2022-10-24 06:09:36 +03:00
Anna Shaleva
6b216050f3 rpc: add compat tests for RPC* interfaces and Client implementations 2022-10-24 06:09:36 +03:00
Anna Shaleva
00d44235c1 rpc: add tests for RPC waiters 2022-10-24 06:09:36 +03:00
Anna Shaleva
388112dcaa rpc: mark old WSClient's SubscribeFor* methods as deprecated 2022-10-24 06:09:36 +03:00
Anna Shaleva
5b81cb065f rpc: refactor waiter-related actor code 2022-10-24 06:09:36 +03:00
Anna Shaleva
8e84bb51d5 rpc: add "till" filter to WS block events 2022-10-24 06:09:36 +03:00
Anna Shaleva
345d48d051 rpc: improve WS-based transaction awaiting
Subscribe only for required aers.
2022-10-24 06:09:36 +03:00
Anna Shaleva
d7c1f3eac7 rpc: add "container" filter to WS execution notifications 2022-10-24 06:09:36 +03:00
Anna Shaleva
71069b0ed0 rpc: improve WS-enabled transaction awaiting
Fetch blocks since VUB+1, if block received and we haven't returned the
result yet, then transaction wasn't accepted at all.
2022-10-24 06:09:36 +03:00
Anna Shaleva
673a495527 rpc: add "since" filter to WS block events 2022-10-24 06:09:36 +03:00
Anna Shaleva
10a0716217 rpc: implement transaction awaiting functionality
Close #2704.
2022-10-24 06:09:36 +03:00
Anna Shaleva
6d38e75149 rpc: support multiple WSClient notification receivers 2022-10-24 06:09:36 +03:00
Anna Shaleva
4ce6bc6a66 rpc: adjust comment to Notification value cast
And explicitly specify field names on Notification creation.
2022-10-24 06:09:36 +03:00
Roman Khimov
0c3b03617e core: optimize HasBlock check for recent blocks
When block is being spread through the network we can get a lot of invs with
the same hash. Some more stale nodes may also announce previous or some
earlier block. We can avoid full DB lookup for them and minimize inv handling
time (timeouts in inv handler had happened in #2744).

It doesn't affect tests, just makes node a little less likely to spend some
considerable amount of time in the inv handler.
2022-10-21 21:21:58 +03:00
Roman Khimov
e26055190e network: check inv against currently processed transactions
Sometimes we already have it, but it's not yet processed, so we can save on
getdata request. It only affects very high-speed networks like 4-1 scenario
and it doesn't affect it a lot, but still we can do it.
2022-10-21 21:16:18 +03:00
Roman Khimov
cfb5058018 network: batch getdata replies
This is not exactly the protocol-level batching as was tried in #1770 and
proposed by neo-project/neo#2365, but it's a TCP-level change in that we now
Write() a set of messages and given that Go sets up TCP sockets with
TCP_NODELAY by default this is a substantial change, we have less packets
generated with the same amount of data. It doesn't change anything on properly
connected networks, but the ones with delays benefit from it a lot.

This also improves queueing because we no longer generate 32 messages to
deliver on transaction's GetData, it's just one stream of bytes with 32
messages inside.

Do the same with GetBlocksByIndex, we can have a lot of messages there too.

But don't forget about potential peer DoS attacks, if a peer is to request a
lot of big blocks we need to flush them before we process the whole set.
2022-10-21 17:16:32 +03:00
Roman Khimov
bf4636f70a
Merge pull request #2755 from nspcc-dev/improve-networking
Improve P2P transaction handling
2022-10-21 16:22:02 +07:00
Roman Khimov
e1b5ac9b81 network: separate tx handling from msg handling
This allows to naturally scale transaction processing if we have some peer
that is sending a lot of them while others are mostly silent. It also can help
somewhat in the event we have 50 peers that all send transactions. 4+1
scenario benefits a lot from it, while 7+2 slows down a little. Delayed
scenarios don't care.

Surprisingly, this also makes disconnects (#2744) much more rare, 4-node
scenario almost never sees it now. Most probably this is the case where peers
affect each other a lot, single-threaded transaction receiver can be slow
enough to trigger some timeout in getdata handler of its peer (because it
tries to push a number of replies).
2022-10-21 12:11:24 +03:00
Roman Khimov
e003b67418 network: reuse inventory hash list for request hashes
Microoptimization, we can do this because we only use them in handleInvCmd().
2022-10-21 11:28:40 +03:00
Roman Khimov
493b979b95 dbft: rev up, fix context timestamp initialization
See nspcc-dev/dbft#64, fixes #2753.
2022-10-20 12:38:14 +03:00
Roman Khimov
0f625f04f0
Merge pull request #2748 from nspcc-dev/stop-tx-flow
network/consensus: use new dbft StopTxFlow callback
2022-10-18 16:29:37 +07:00
Roman Khimov
73ce898e27 network/consensus: use new dbft StopTxFlow callback
It makes sense in general (further narrowing down the time window when
transactions are processed by consensus thread) and it improves block times a
little too, especially in the 7+2 scenario.

Related to #2744.
2022-10-18 11:06:20 +03:00
Roman Khimov
2791127ee4 network: add prometheus histogram with cmd processing time
It can be useful to detect some performance issues.
2022-10-17 22:51:16 +03:00
Roman Khimov
73079745ab
Merge pull request #2746 from nspcc-dev/optimize-tx-callbacks
network: only call tx callback if we're waiting for transactions
2022-10-17 16:39:41 +07:00
Roman Khimov
dce9f80585
Merge pull request #2743 from nspcc-dev/log-fan-out
Logarithmic gossip fan out
2022-10-14 23:18:34 +07:00
Roman Khimov
4dd3fd4ac0 network: only call tx callback if we're waiting for transactions
Until the consensus process starts for a new block and until it really needs
some transactions we can spare some cycles by not delivering transactions to
it. In tests this doesn't affect TPS, but makes block delays a bit more
stable. Related to #2744, I think it also may cause timeouts during
transaction processing (waiting on the consensus process channel while it does
something dBFT-related).
2022-10-14 18:45:48 +03:00
Roman Khimov
65f0fadddb network: register peer only if it's not a duplicate 2022-10-14 15:53:32 +03:00
Roman Khimov
851cbc7dab network: implement adaptive peer requests
When the network is big enough, MinPeers may be suboptimal for good network
connectivity, but if we know the network size we can do some estimation on the
number of sufficient peers.
2022-10-14 15:53:32 +03:00
Roman Khimov
c17b2afab5 network: add BroadcastFactor to control gossip, fix #2678 2022-10-14 15:53:32 +03:00
Roman Khimov
215e8704f1 network: simplify discoverer, make it almost a lib
We already have two basic lists: connected and unconnected nodes, we don't
need an additional channel and we don't need a goroutine to handle it.
2022-10-14 15:53:32 +03:00
Roman Khimov
c1ef326183 network: re-add addresses to the pool on UnregisterConnectedAddr
That's what we do anyway, but this way we can be a bit more efficient.
2022-10-14 14:12:33 +03:00
Roman Khimov
631f166709 network: broadcast to log-dependent number of nodes
Fixes #608.
2022-10-14 14:12:33 +03:00
Roman Khimov
c3001bc5bd
Merge pull request #2740 from nspcc-dev/cli-improvement
cli: improve VM CLI a bit more
2022-10-13 20:21:47 +07:00
Anna Shaleva
7db9258104 vm: make LoadFileWithFlags actually load with flags provided 2022-10-13 16:07:34 +03:00
Anna Shaleva
4dbaf2a123 smartcontract: add comment to GetCompleteTransaction 2022-10-13 16:07:34 +03:00
Anna Shaleva
af658bc3e5 cli: support Null as an argument for invocation-related commands 2022-10-13 16:07:31 +03:00
Anna Shaleva
7eb87afab8 cli: unify parameters parsing
Share parameters parsing code between 'contract invokefunction' and
'vm run' commands. It allows VM CLI to parse more complicated parameter
types including arrays and file-backed bytestrings.
2022-10-13 08:20:27 +03:00