Commit graph

4646 commits

Author SHA1 Message Date
Roman Khimov
bcf77c3c42 network: filter out not-yet-ready nodes when broadcasting
They can fail right in the getPeers or they can fail later when packet send
is attempted. Of course they can complete handshake in-between these events,
but most likely they won't and we'll waste more resources on this attempt. So
rule out bad peers immediately.
2022-10-12 16:51:01 +03:00
Roman Khimov
137f2cb192 network: deduplicate TCPPeer code a bit
context.Background() is never canceled and has no deadline, so we can avoid
duplicating some code.
2022-10-12 15:43:31 +03:00
Roman Khimov
104da8caff network: broadcast messages, enqueue packets
Drop EnqueueP2PPacket, replace EnqueueHPPacket with EnqueueHPMessage. We use
Enqueue* when we have a specific per-peer message, it makes zero sense
duplicating serialization code for it (unlike Broadcast*).
2022-10-12 15:39:20 +03:00
Roman Khimov
d5f2ad86a1 network: drop unused EnqueueMessage interface from Peer 2022-10-12 15:27:08 +03:00
Roman Khimov
b345581c72 network: pings are broadcasted, don't send them to everyone
Follow the general rules of broadcasts, even though it's somewhat different
from Inv, we just want to get some reply from our neighbors to see if we're
behind. We don't strictly need all neighbors for it.
2022-10-12 15:25:03 +03:00
Roman Khimov
e1d5f18ff4 network: fix outdated Peer interface comments 2022-10-12 10:16:07 +03:00
Roman Khimov
8b26d9475b network: speculatively set GetAddrSent status
Otherwise we routinely get "unexpected addr received" error.
2022-10-11 18:42:40 +03:00
Roman Khimov
e80c60a3b9 network: rework broadcast logic
We have a number of queues for different purposes:
 * regular broadcast queue
 * direct p2p queue
 * high-priority queue

And two basic egress scenarios:
 * direct p2p messages (replies to requests in Server's handle* methods)
 * broadcasted messages

Low priority broadcasted messages:
 * transaction inventories
 * block inventories
 * notary inventories
 * non-consensus extensibles

High-priority broadcasted messages:
 * consensus extensibles
 * getdata transaction requests from consensus process
 * getaddr requests

P2P messages are a bit more complicated, most of the time they use p2p queue,
but extensible message requests/replies use HP queue.

Server's handle* code is run from Peer's handleIncoming, every peer has this
thread that handles incoming messages. When working with the peer it's
important to reply to requests and blocking this thread until we send (queue)
a reply is fine, if the peer is slow we just won't get anything new from
it. The queue used is irrelevant wrt this issue.

Broadcasted messages are radically different, we want them to be delivered to
many peers, but we don't care about specific ones. If it's delivered to 2/3 of
the peers we're fine, if it's delivered to more of them --- it's not an
issue. But doing this fairly is not an easy thing, current code tries performing
unblocked sends and if this doesn't yield enough results it then blocks (but
has a timeout, we can't wait indefinitely). But it does so in sequential
manner, once the peer is chosen the code will wait for it (and only it) until
timeout happens.

What can be done instead is an attempt to push the message to all of the peers
simultaneously (or close to that). If they all deliver --- OK, if some block
and wait then we can wait until _any_ of them pushes the message through (or
global timeout happens, we still can't wait forever). If we have enough
deliveries then we can cancel pending ones and it's again not an error if
these canceled threads still do their job.

This makes the system more dynamic and adds some substantial processing
overhead, but it's a networking code, any of this overhead is much lower than
the actual packet delivery time. It also allows to spread the load more
fairly, if there is any spare queue it'll get the packet and release the
broadcaster. On the next broadcast iteration another peer is more likely to be
chosen just because it didn't get a message previously (and had some time to
deliver already queued messages).

It works perfectly in tests, with optimal networking conditions we have much
better block times and TPS increases by 5-25%% depending on the scenario.

I'd go as far as to say that it fixes the original problem of #2678, because
in this particular scenario we have empty queues in ~100% of the cases and
this new logic will likely lead to 100% fan out in this case (cancelation just
won't happen fast enough). But when the load grows and there is some waiting
in the queue it will optimize out the slowest links.
2022-10-11 18:42:40 +03:00
Roman Khimov
0294e2eb18
Merge pull request #2738 from nspcc-dev/dont-block-forever-2
network: don't wait indefinitely for packet to be sent
2022-10-11 19:40:10 +07:00
Anna Shaleva
641abd4d1c smartcontract: fix underlying PublicKey parameter value
Value of PublicKey parameter always stores public key bytes, not the
deserialized representation. All other code (CLI parameters parsing with
its NewParameterFromString, Parameter unmarshaller, etc.) is based on
the idea that value of PublicKey is []byte.
2022-10-11 13:50:32 +03:00
Roman Khimov
dabdad20ad network: don't wait indefinitely for packet to be sent
Peers can be slow, very slow, slow enough to affect node's regular
operation. We can't wait for them indefinitely, there has to be a timeout for
send operations.

This patch uses TimePerBlock as a reference for its timeout. It's relatively
big and it doesn't affect tests much, 4+1 scenarios tend to perform a little
worse with while 7+2 scenarios work a little better. The difference is in some
percents, but all of these tests easily have 10-15% variations from run to
run.

It's an important step in making our gossip better because we can't have any
behavior where neighbors directly block the node forever, refs. #2678 and
2022-10-10 22:15:21 +03:00
Anna Shaleva
63fddb3f1a core: close BoltDB on failed root bucket creation 2022-10-10 10:12:34 +03:00
Anna Shaleva
735db08f84 services: adjust RPC server's getHistoricParams
Update documentation and add index upper bound check to get rid of
CodeQL warning.
2022-10-07 16:06:12 +03:00
Anna Shaleva
95cbddf19e cli: use custom logger to filter out runtime.Log messages
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go vm -p

    _   ____________        __________      _    ____  ___
   / | / / ____/ __ \      / ____/ __ \    | |  / /  |/  /
  /  |/ / __/ / / / /_____/ / __/ / / /____| | / / /|_/ /
 / /|  / /___/ /_/ /_____/ /_/ / /_/ /_____/ |/ / /  / /
/_/ |_/_____/\____/      \____/\____/      |___/_/  /_/

NEO-GO-VM > loadgo ./1-print/1-print.go
READY: loaded 21 instructions
NEO-GO-VM 0 > run
2022-10-07T15:28:20.461+0300	INFO	runtime log	{"tx": "", "script": "db03ceb3f672ee8cd0d714989b4d103ff7eed2f3", "msg": "Hello, world!"}
[]
```
2022-10-07 15:57:33 +03:00
Anna Shaleva
0db4e8d62c core: allow to perform storage search within given amount of DAO layers 2022-10-07 15:56:34 +03:00
Anna Shaleva
79e13f73d8 core, rpc: move getFakeNextBlock to Blockchain
It's needed for VM CLI as far and may be improved later.
2022-10-07 15:56:34 +03:00
Anna Shaleva
f45d8fc08d vm: remove default syscall handler
It's not needed anymore. Close #1075.
2022-10-07 15:56:34 +03:00
Anna Shaleva
0b717b0c22 vm: move vm CLI to cli/vm package 2022-10-07 15:56:34 +03:00
Anna Shaleva
4a46001746 smartcontract: fix error message for CreateMultiSigRedeemScript 2022-10-07 15:56:34 +03:00
Anna Shaleva
a91cf2a007 core: set default SecondsPerBlock value on blockchain creation
As mentioned in the node configuration docs.
2022-10-07 15:56:34 +03:00
Anna Shaleva
70e59d83c9 docs: fix supported database types 2022-10-07 15:56:34 +03:00
Anna Shaleva
2f5137e9b7 core: allow RO mode for Bolt and Level 2022-10-07 15:56:29 +03:00
Anna Shaleva
cbdd45cc96 core: return error on root BoltDB bucket creation if so 2022-10-06 14:01:56 +03:00
Anna Shaleva
03a1cf9f59 core: simplify newLevelDBForTesting function 2022-10-06 14:01:56 +03:00
Roman Khimov
4616600636
Merge pull request #2728 from nspcc-dev/fix-vub-comment
core: add example to VUB comment
2022-10-06 16:33:24 +07:00
Anna Shaleva
78cd2b4566 core: add example to VUB comment 2022-10-06 10:32:18 +03:00
Roman Khimov
1c99c21d9a
Merge pull request #2725 from nspcc-dev/move-cli-tests
Move CLI tests
2022-10-06 13:43:39 +07:00
Roman Khimov
1ac60ada19 cli: move tests to subpackages
Refs. #2379, but not completely solves it, one package seriously outweights
others:

?       github.com/nspcc-dev/neo-go/cli [no test files]
ok      github.com/nspcc-dev/neo-go/cli/app     0.036s  coverage: 100.0% of statements
ok      github.com/nspcc-dev/neo-go/cli/cmdargs 0.011s  coverage: 60.8% of statements
ok      github.com/nspcc-dev/neo-go/cli/flags   0.009s  coverage: 97.7% of statements
?       github.com/nspcc-dev/neo-go/cli/input   [no test files]
ok      github.com/nspcc-dev/neo-go/cli/options 0.033s  coverage: 50.0% of statements
?       github.com/nspcc-dev/neo-go/cli/paramcontext    [no test files]
ok      github.com/nspcc-dev/neo-go/cli/query   2.155s  coverage: 45.3% of statements
ok      github.com/nspcc-dev/neo-go/cli/server  1.373s  coverage: 67.8% of statements
ok      github.com/nspcc-dev/neo-go/cli/smartcontract   8.819s  coverage: 94.3% of statements
ok      github.com/nspcc-dev/neo-go/cli/util    0.006s  coverage: 10.9% of statements
?       github.com/nspcc-dev/neo-go/cli/vm      [no test files]
ok      github.com/nspcc-dev/neo-go/cli/wallet  72.103s coverage: 88.2% of statements

Still a nice thing to have.
2022-10-06 09:21:26 +03:00
Roman Khimov
1c376ffa62
Merge pull request #2724 from nspcc-dev/rpc-options
rpcsrv: handle preflight OPTIONS with CORS kludge, fix #2721
2022-10-05 17:02:55 +07:00
Roman Khimov
b48d02f4a6 rpcsrv: handle preflight OPTIONS with CORS kludge, fix #2721 2022-10-05 11:09:45 +03:00
Roman Khimov
8893163803 smartcontract: define parameter lengths as constants and use them 2022-10-05 10:46:21 +03:00
Roman Khimov
317dd42513 *: use uint*Size and SignatureLen constants where appropriate 2022-10-05 10:45:52 +03:00
Roman Khimov
79887f9d78 runtime: check notifications against ABI
Related to #2703, just a logged thing for now.
2022-10-04 17:52:38 +03:00
Roman Khimov
7d0840d5d5
Merge pull request #2720 from nspcc-dev/notifications-check
compiler: enforce runtime.Notify parameters cast to proper type
2022-10-01 03:02:29 +07:00
Anna Shaleva
554e48e7b7 compiler: enforce runtime.Notify parameters cast
If notification parameters type can be defined in a compile time then enforce
parameter cast to the desired type got from manifest.
2022-09-30 14:42:43 +03:00
Anna Shaleva
80f71a4e6e compiler: do not enforce variadic event args check on ellipsis usage
In case of ellipsis usage compiler defines argument type as ArrayT
(which is correct, because it's a natural representation of the last
argument, it represents the array of interface{}).
Here goes the problem:
```
=== RUN   TestEventWarnings/variadic_event_args_via_ellipsis
    compiler_test.go:251:
        	Error Trace:	compiler_test.go:251
        	Error:      	Received unexpected error:
        	            	event 'Event' should have 'Integer' as type of 1 parameter, got: Array
        	Test:       	TestEventWarnings/variadic_event_args_via_ellipsis
```

Parsing the last argument in this case is a separate complicated problem
due to the fact that we need to grab types of elements of []interface{} inside the
fully qualified ast node which may looks like:
```
runtime.Notify("Event", (append([]interface{}{1, 2}, (([]interface{}{someVar, 4}))...))...)
```

Temporary solution is to exclude such notifications from analysis until we're
able to properly resolve element types of []interface{}.
2022-09-30 08:42:48 +03:00
Anna Shaleva
08427f23b6 compiler: do not check Any event parameter for compliance
It's possible that declared manifest event has parameter of AnyT for
those cases when parameter type differs from method to method. If so,
then we don't need to enforce type check after compilation.
2022-09-30 08:40:55 +03:00
Roman Khimov
b7be4edf7f
Merge pull request #2718 from nspcc-dev/ok-conversion
compiler: prohibit to compile type assertion with two return values
2022-09-29 02:54:03 +07:00
Anna Shaleva
b98848bf49 compiler: prohibit to compile type assertion with two return values
Close #2692.
2022-09-28 11:27:13 +03:00
Anna Shaleva
1828e79412 compiler: add test for foreign function inlining 2022-09-27 15:36:06 +03:00
Anna Shaleva
24c107e3a2 interop: refactor address.FromHash160 code
Make it more simple.
2022-09-27 15:32:57 +03:00
Anna Shaleva
5d578fdd95 compiler: consider inlined types info on "append" handling
We need to search for "append" argument type info not only inside local
package type info map, but also inside the inlined type info map.
Close #2696.
2022-09-27 15:32:51 +03:00
Anna Shaleva
ea08a81726 interop: add CallWithVersion helper 2022-09-21 17:38:42 +03:00
Anna Shaleva
df802b6fc6 vm: adjust emit.AppCall comment
There's no APPCALL anymore.
2022-09-21 17:25:45 +03:00
Anna Shaleva
8d5f97a699 interop: adjust Iterator documentation
The only way to get Iterator is as a result of storage.Find.
2022-09-20 17:09:48 +03:00
Roman Khimov
5a7ab2054d
Merge pull request #2698 from nspcc-dev/address-helpers
interop: add a couple of `interop.Hash160` encoding helpers
2022-09-20 17:07:00 +07:00
Anna Shaleva
293dbf3d1b compiler: adjust test's checkInstrCount
Remove unnecessary code.
2022-09-20 09:45:07 +03:00
Anna Shaleva
7e13140b04 interop: add Hash160 encoder\decoder helper
Close #2690.
2022-09-20 09:37:04 +03:00
Roman Khimov
1b753cd4bc native: add some tests for stdlib's atoi
See neo-project/neo#2804 and neo-project/neo#2813. We're already compatible.
2022-09-19 16:18:53 +03:00
Roman Khimov
18ed26194f smartcontract: add Len to Builder
Which is useful in some cases.
2022-09-14 10:25:10 +03:00