Commit graph

6398 commits

Author SHA1 Message Date
Roman Khimov
c1ef326183 network: re-add addresses to the pool on UnregisterConnectedAddr
That's what we do anyway, but this way we can be a bit more efficient.
2022-10-14 14:12:33 +03:00
Roman Khimov
631f166709 network: broadcast to log-dependent number of nodes
Fixes #608.
2022-10-14 14:12:33 +03:00
Roman Khimov
dc62046019 network: add network size estimation metric 2022-10-12 22:29:55 +03:00
Roman Khimov
ec4983e88e
Merge pull request #2741 from nspcc-dev/separate-broadcast-queue-handling
Rework broadcast logic
2022-10-12 16:33:27 +07:00
Roman Khimov
e1d5f18ff4 network: fix outdated Peer interface comments 2022-10-12 10:16:07 +03:00
Roman Khimov
8b26d9475b network: speculatively set GetAddrSent status
Otherwise we routinely get "unexpected addr received" error.
2022-10-11 18:42:40 +03:00
Roman Khimov
e80c60a3b9 network: rework broadcast logic
We have a number of queues for different purposes:
 * regular broadcast queue
 * direct p2p queue
 * high-priority queue

And two basic egress scenarios:
 * direct p2p messages (replies to requests in Server's handle* methods)
 * broadcasted messages

Low priority broadcasted messages:
 * transaction inventories
 * block inventories
 * notary inventories
 * non-consensus extensibles

High-priority broadcasted messages:
 * consensus extensibles
 * getdata transaction requests from consensus process
 * getaddr requests

P2P messages are a bit more complicated, most of the time they use p2p queue,
but extensible message requests/replies use HP queue.

Server's handle* code is run from Peer's handleIncoming, every peer has this
thread that handles incoming messages. When working with the peer it's
important to reply to requests and blocking this thread until we send (queue)
a reply is fine, if the peer is slow we just won't get anything new from
it. The queue used is irrelevant wrt this issue.

Broadcasted messages are radically different, we want them to be delivered to
many peers, but we don't care about specific ones. If it's delivered to 2/3 of
the peers we're fine, if it's delivered to more of them --- it's not an
issue. But doing this fairly is not an easy thing, current code tries performing
unblocked sends and if this doesn't yield enough results it then blocks (but
has a timeout, we can't wait indefinitely). But it does so in sequential
manner, once the peer is chosen the code will wait for it (and only it) until
timeout happens.

What can be done instead is an attempt to push the message to all of the peers
simultaneously (or close to that). If they all deliver --- OK, if some block
and wait then we can wait until _any_ of them pushes the message through (or
global timeout happens, we still can't wait forever). If we have enough
deliveries then we can cancel pending ones and it's again not an error if
these canceled threads still do their job.

This makes the system more dynamic and adds some substantial processing
overhead, but it's a networking code, any of this overhead is much lower than
the actual packet delivery time. It also allows to spread the load more
fairly, if there is any spare queue it'll get the packet and release the
broadcaster. On the next broadcast iteration another peer is more likely to be
chosen just because it didn't get a message previously (and had some time to
deliver already queued messages).

It works perfectly in tests, with optimal networking conditions we have much
better block times and TPS increases by 5-25%% depending on the scenario.

I'd go as far as to say that it fixes the original problem of #2678, because
in this particular scenario we have empty queues in ~100% of the cases and
this new logic will likely lead to 100% fan out in this case (cancelation just
won't happen fast enough). But when the load grows and there is some waiting
in the queue it will optimize out the slowest links.
2022-10-11 18:42:40 +03:00
Roman Khimov
0294e2eb18
Merge pull request #2738 from nspcc-dev/dont-block-forever-2
network: don't wait indefinitely for packet to be sent
2022-10-11 19:40:10 +07:00
Roman Khimov
44df4b9dbb
Merge pull request #2739 from nspcc-dev/fix-sc-conversion
smartcontract: fix underlying PublicKey parameter value
2022-10-11 18:06:19 +07:00
Anna Shaleva
641abd4d1c smartcontract: fix underlying PublicKey parameter value
Value of PublicKey parameter always stores public key bytes, not the
deserialized representation. All other code (CLI parameters parsing with
its NewParameterFromString, Parameter unmarshaller, etc.) is based on
the idea that value of PublicKey is []byte.
2022-10-11 13:50:32 +03:00
Roman Khimov
dabdad20ad network: don't wait indefinitely for packet to be sent
Peers can be slow, very slow, slow enough to affect node's regular
operation. We can't wait for them indefinitely, there has to be a timeout for
send operations.

This patch uses TimePerBlock as a reference for its timeout. It's relatively
big and it doesn't affect tests much, 4+1 scenarios tend to perform a little
worse with while 7+2 scenarios work a little better. The difference is in some
percents, but all of these tests easily have 10-15% variations from run to
run.

It's an important step in making our gossip better because we can't have any
behavior where neighbors directly block the node forever, refs. #2678 and
2022-10-10 22:15:21 +03:00
Roman Khimov
478b4b0c1c
Merge pull request #2737 from nspcc-dev/fix-win-cleanup
cli: properly cleanup state-dependand VM CLI
2022-10-10 15:49:18 +07:00
Anna Shaleva
63fddb3f1a core: close BoltDB on failed root bucket creation 2022-10-10 10:12:34 +03:00
Anna Shaleva
343ce4d4bb cli: properly cleanup state-dependand VM CLI
Close #2736.
2022-10-10 08:26:22 +03:00
Roman Khimov
f86cdd95b2
Merge pull request #2735 from nspcc-dev/changelog-0.99.4
CHANGELOG: 0.99.4
2022-10-07 21:23:48 +07:00
Roman Khimov
1426b54fd7
Merge pull request #2723 from nspcc-dev/smart-vm-cli
vm: make VM CLI state-dependant
2022-10-07 21:21:11 +07:00
Roman Khimov
4d2afa2624 cli/vm: use ParseInt to properly (and easily) check for int32 2022-10-07 17:10:04 +03:00
Anna Shaleva
735db08f84 services: adjust RPC server's getHistoricParams
Update documentation and add index upper bound check to get rid of
CodeQL warning.
2022-10-07 16:06:12 +03:00
Anna Shaleva
eac5e1526e cli: rename VMCLI to CLI 2022-10-07 15:57:33 +03:00
Anna Shaleva
16f5ae3812 cli: add upper bound check for contract ID for 'storage' VM CLI cmd 2022-10-07 15:57:33 +03:00
Anna Shaleva
95cbddf19e cli: use custom logger to filter out runtime.Log messages
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go vm -p

    _   ____________        __________      _    ____  ___
   / | / / ____/ __ \      / ____/ __ \    | |  / /  |/  /
  /  |/ / __/ / / / /_____/ / __/ / / /____| | / / /|_/ /
 / /|  / /___/ /_/ /_____/ /_/ / /_/ /_____/ |/ / /  / /
/_/ |_/_____/\____/      \____/\____/      |___/_/  /_/

NEO-GO-VM > loadgo ./1-print/1-print.go
READY: loaded 21 instructions
NEO-GO-VM 0 > run
2022-10-07T15:28:20.461+0300	INFO	runtime log	{"tx": "", "script": "db03ceb3f672ee8cd0d714989b4d103ff7eed2f3", "msg": "Hello, world!"}
[]
```
2022-10-07 15:57:33 +03:00
Anna Shaleva
b2cd007d8d cli: move handleLoggingParams to 'options' package
It will be reused by other CLI packages.
2022-10-07 15:57:33 +03:00
Anna Shaleva
b3c8192d2e cli: add 'changes' command for VM CLI 2022-10-07 15:57:31 +03:00
Anna Shaleva
cac4f6a4a6 cli: allow to dump storage diff for 'storage' VM CLI command 2022-10-07 15:56:34 +03:00
Anna Shaleva
0db4e8d62c core: allow to perform storage search within given amount of DAO layers 2022-10-07 15:56:34 +03:00
Anna Shaleva
8c78177806 vm: add 'storage' VM CLI command to dump storage items
Another nice one, very useful for debugging.
2022-10-07 15:56:34 +03:00
Anna Shaleva
ff03c33e6d vm: allow historic calls inside VM CLI 2022-10-07 15:56:34 +03:00
Anna Shaleva
79e13f73d8 core, rpc: move getFakeNextBlock to Blockchain
It's needed for VM CLI as far and may be improved later.
2022-10-07 15:56:34 +03:00
Anna Shaleva
0036c89d63 vm: add 'env' command showing state of the blockchain-backed VM CLI
A useful one.
2022-10-07 15:56:34 +03:00
Anna Shaleva
f1ecdb82cc vm: add 'events' command to VM CLI
And dump events automatically after HALTed or FAULTed end of execution.
2022-10-07 15:56:34 +03:00
Anna Shaleva
f45d8fc08d vm: remove default syscall handler
It's not needed anymore. Close #1075.
2022-10-07 15:56:34 +03:00
Anna Shaleva
33ae8d0ddc vm: clear manifest on VM CLI reset
Fix the bug when outdated manifest was stored after new program was loaded.
2022-10-07 15:56:34 +03:00
Anna Shaleva
513821cfff vm: allow to provide state for VM CLI
Close #2528.

Also, add new simple testchain as an analogue for basicchain.
2022-10-07 15:56:34 +03:00
Anna Shaleva
0b717b0c22 vm: move vm CLI to cli/vm package 2022-10-07 15:56:34 +03:00
Anna Shaleva
4a46001746 smartcontract: fix error message for CreateMultiSigRedeemScript 2022-10-07 15:56:34 +03:00
Anna Shaleva
a91cf2a007 core: set default SecondsPerBlock value on blockchain creation
As mentioned in the node configuration docs.
2022-10-07 15:56:34 +03:00
Anna Shaleva
70e59d83c9 docs: fix supported database types 2022-10-07 15:56:34 +03:00
Anna Shaleva
2f5137e9b7 core: allow RO mode for Bolt and Level 2022-10-07 15:56:29 +03:00
Roman Khimov
d93f3f416a CHANGELOG: 0.99.4 2022-10-07 12:10:15 +03:00
Roman Khimov
da70b9e5ec
Merge pull request #2730 from nspcc-dev/neofs-configs
config: add NeoFS network configurations, fix #2714
2022-10-07 12:51:43 +07:00
Roman Khimov
5c94ffab37
Merge pull request #2731 from nspcc-dev/cli-tx-outs
Add ability to save any tx in the CLI
2022-10-07 12:47:17 +07:00
Roman Khimov
0efe3dd42c cli: deduplicate smartcontract/wallet tx confirm/sign/save/send
It's the same code.
2022-10-06 23:03:32 +03:00
Roman Khimov
5200765dab cli/wallet: add out/force flags to claim/candidate cmds
Fixes #2669.
2022-10-06 22:19:40 +03:00
Roman Khimov
5bafa5f6b4 cli/wallet: add sysgas option to candidate and claim commands
It doesn't make a lot of sense, but hey, it's a transaction, it can have a bit
more GAS.
2022-10-06 22:09:32 +03:00
Roman Khimov
1ea06e9757 cli/wallet: unify claim and candidate code
Add -g option to claim which doesn't have a lot of sense, but can be used
anyway.
2022-10-06 21:52:40 +03:00
Roman Khimov
d9f3ff3969 config: add NeoFS network configurations, fix #2714 2022-10-06 18:57:04 +03:00
Anna Shaleva
cbdd45cc96 core: return error on root BoltDB bucket creation if so 2022-10-06 14:01:56 +03:00
Anna Shaleva
03a1cf9f59 core: simplify newLevelDBForTesting function 2022-10-06 14:01:56 +03:00
Anna Shaleva
c4c93b591e cli: remove debug flag from VM CLI
It's not used.
2022-10-06 14:01:56 +03:00
Anna Shaleva
5698ce03be cli: move debug flag to options package 2022-10-06 14:01:56 +03:00