We have a number of queues for different purposes:
* regular broadcast queue
* direct p2p queue
* high-priority queue
And two basic egress scenarios:
* direct p2p messages (replies to requests in Server's handle* methods)
* broadcasted messages
Low priority broadcasted messages:
* transaction inventories
* block inventories
* notary inventories
* non-consensus extensibles
High-priority broadcasted messages:
* consensus extensibles
* getdata transaction requests from consensus process
* getaddr requests
P2P messages are a bit more complicated, most of the time they use p2p queue,
but extensible message requests/replies use HP queue.
Server's handle* code is run from Peer's handleIncoming, every peer has this
thread that handles incoming messages. When working with the peer it's
important to reply to requests and blocking this thread until we send (queue)
a reply is fine, if the peer is slow we just won't get anything new from
it. The queue used is irrelevant wrt this issue.
Broadcasted messages are radically different, we want them to be delivered to
many peers, but we don't care about specific ones. If it's delivered to 2/3 of
the peers we're fine, if it's delivered to more of them --- it's not an
issue. But doing this fairly is not an easy thing, current code tries performing
unblocked sends and if this doesn't yield enough results it then blocks (but
has a timeout, we can't wait indefinitely). But it does so in sequential
manner, once the peer is chosen the code will wait for it (and only it) until
timeout happens.
What can be done instead is an attempt to push the message to all of the peers
simultaneously (or close to that). If they all deliver --- OK, if some block
and wait then we can wait until _any_ of them pushes the message through (or
global timeout happens, we still can't wait forever). If we have enough
deliveries then we can cancel pending ones and it's again not an error if
these canceled threads still do their job.
This makes the system more dynamic and adds some substantial processing
overhead, but it's a networking code, any of this overhead is much lower than
the actual packet delivery time. It also allows to spread the load more
fairly, if there is any spare queue it'll get the packet and release the
broadcaster. On the next broadcast iteration another peer is more likely to be
chosen just because it didn't get a message previously (and had some time to
deliver already queued messages).
It works perfectly in tests, with optimal networking conditions we have much
better block times and TPS increases by 5-25%% depending on the scenario.
I'd go as far as to say that it fixes the original problem of #2678, because
in this particular scenario we have empty queues in ~100% of the cases and
this new logic will likely lead to 100% fan out in this case (cancelation just
won't happen fast enough). But when the load grows and there is some waiting
in the queue it will optimize out the slowest links.
Value of PublicKey parameter always stores public key bytes, not the
deserialized representation. All other code (CLI parameters parsing with
its NewParameterFromString, Parameter unmarshaller, etc.) is based on
the idea that value of PublicKey is []byte.
Peers can be slow, very slow, slow enough to affect node's regular
operation. We can't wait for them indefinitely, there has to be a timeout for
send operations.
This patch uses TimePerBlock as a reference for its timeout. It's relatively
big and it doesn't affect tests much, 4+1 scenarios tend to perform a little
worse with while 7+2 scenarios work a little better. The difference is in some
percents, but all of these tests easily have 10-15% variations from run to
run.
It's an important step in making our gossip better because we can't have any
behavior where neighbors directly block the node forever, refs. #2678 and
Refs. #2379, but not completely solves it, one package seriously outweights
others:
? github.com/nspcc-dev/neo-go/cli [no test files]
ok github.com/nspcc-dev/neo-go/cli/app 0.036s coverage: 100.0% of statements
ok github.com/nspcc-dev/neo-go/cli/cmdargs 0.011s coverage: 60.8% of statements
ok github.com/nspcc-dev/neo-go/cli/flags 0.009s coverage: 97.7% of statements
? github.com/nspcc-dev/neo-go/cli/input [no test files]
ok github.com/nspcc-dev/neo-go/cli/options 0.033s coverage: 50.0% of statements
? github.com/nspcc-dev/neo-go/cli/paramcontext [no test files]
ok github.com/nspcc-dev/neo-go/cli/query 2.155s coverage: 45.3% of statements
ok github.com/nspcc-dev/neo-go/cli/server 1.373s coverage: 67.8% of statements
ok github.com/nspcc-dev/neo-go/cli/smartcontract 8.819s coverage: 94.3% of statements
ok github.com/nspcc-dev/neo-go/cli/util 0.006s coverage: 10.9% of statements
? github.com/nspcc-dev/neo-go/cli/vm [no test files]
ok github.com/nspcc-dev/neo-go/cli/wallet 72.103s coverage: 88.2% of statements
Still a nice thing to have.
In case of ellipsis usage compiler defines argument type as ArrayT
(which is correct, because it's a natural representation of the last
argument, it represents the array of interface{}).
Here goes the problem:
```
=== RUN TestEventWarnings/variadic_event_args_via_ellipsis
compiler_test.go:251:
Error Trace: compiler_test.go:251
Error: Received unexpected error:
event 'Event' should have 'Integer' as type of 1 parameter, got: Array
Test: TestEventWarnings/variadic_event_args_via_ellipsis
```
Parsing the last argument in this case is a separate complicated problem
due to the fact that we need to grab types of elements of []interface{} inside the
fully qualified ast node which may looks like:
```
runtime.Notify("Event", (append([]interface{}{1, 2}, (([]interface{}{someVar, 4}))...))...)
```
Temporary solution is to exclude such notifications from analysis until we're
able to properly resolve element types of []interface{}.
It's possible that declared manifest event has parameter of AnyT for
those cases when parameter type differs from method to method. If so,
then we don't need to enforce type check after compilation.
Which greatly simplifies reuse of these packages (and they're expected to be
reused since real tokens implement standards and also add something of their
own) and allows to avoid effects like
doc_test.go:68:28: ambiguous selector neoContract.BalanceOf
when neo.Contract is used. Avoids duplication in NEP-11 implementation as
well.
So that (*codegen).Visit is able to omit code generation for these
unused global vars. The most tricky part is to detect unused global
variables, it is done in several steps:
1. Collect the set of named used/unused global vars.
2. Collect the set of globally declared expressions that contain
function calls.
3. Pick up global vars from the set made at step 2.
4. Traverse used functions and puck up those global vars that are used
from these functions.
5. Rename all globals that are presented in the set made at step 1
but are not presented in the set made on step 3 or step 4.
Move all auxiliary function declaration after Main, so that INITSLOT
instructions counter works properly. `vmAndCompileInterop` loads program
and moves nextIP to the Main function offset if there's no _init
function. If _init is there, then nextIP will be moved to the start of
_init. In TestInline we don't handle instructions properly (CALL/JMP
don't change nextIP), we just perform instruction traversal from the
start point via Next(), thus INITSLOT counter value depends on the
starting instruction, which depends on _init presence.
If variable is unnamed and does not contain function call then it's
treated as unused and code generation may be omitted for it
initialization/declaration.
In case if global var is unnamed (and, as a consequence, unused) and
contains a function call inside its value specification, we need to emit
code for this var to be able to call the function as it can have
side-effects. See the example:
```
package foo
import "github.com/nspcc-dev/neo-go/pkg/interop/runtime"
var A = f()
func Main() int {
return 3
}
func f() int {
runtime.Notify("Valuable notification", 1)
return 2
}
```
NEP-6 has a notion of locked acccounts and SignTx must respect this user's
choice. For some reason this setting was inappropriately used by our RPC
client tests (probably a different kind of lock was meant).
* each account must have an appropriate signer, if there is no signer for
this account in the tx it's an error
* we can only safely append to Scripts when account belongs to the next
signer (we don't have appropriate verification scripts for other signers)
* when contract has one parameter, the signature shouldn't be appended to
other data
I think these rules allow to handle more cases and do that safer. We have more
complex scenarios though, like non-signature parameters or mixed-parameter
invocation scripts, but that's out of scope for now.
calculatenetworkfee MUST calculate complete proper network fee, if we have
some extensions enabled and some attributes should be paid for that they're a
part of the equation too.
Adding an array multiple times leads to the fast update via `IncRC`.
This hides the allocation that is there on the first addition. In this
commit add another benchmark which measures Add/Remove together, to
ensure that `switch` in `refCounter.Add` is entered. Benchmark results
are meaningful, because `Add`/`Remove` have almost identical implementation.
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
We're dealing with a transaction here and it can't be decoded successfully
unless it has an appropriate number of witness scripts (matching the number of
signers) with appropriate hashes (matching signers). So this iterations make
no sense at all, we know exactly where to look for the
verification/invocation scripts.
Blockchain's notificationDispatcher sends events to channels and these
channels must be read from. Unfortunately, regular service shutdown procedure
does unsubscription first (outside of the read loop) and only then drains the
channel. While it waits for unsubscription request to be accepted
notificationDispatcher can try pushing more data into the same channel which
will lead to a deadlock. Reading in the same method solves this, any number of
events can be pushed until unsub channel accepts the data.
Unsubscribe and drain first, then return from the Shutdown method. It's
important wrt to subsequent chain shutdown process (normally it's closed right
after the network server).
Unfortunately Go doesn't allow to easily reuse readers in full packages, still
we can have this wrapper with a little overhead (the alternative is to move
specific methods into types of their own, but I'm not sure how it's going to
be accepted user-side).
Notice that int64 types are used for gas per block or registration price
because the price has to fit into the system fee limitation and gas per block
value can't be more than 10 GAS. We use int64 for votes as well in other types
since NEO is limited to 100M.
An attempt to compile the following code leads to runtime panic:
```
package foo
type CustomInt int
func Main() int {
var i CustomInt
i = 5
return i.Do(2)
}
func (CustomInt) Do(arg int) int {
return arg
}
```
The panic:
```
panic: runtime error: index out of range [0] with length 0 [recovered]
panic: runtime error: index out of range [0] with length 0
goroutine 22 [running]:
testing.tRunner.func1.2({0xa341c0, 0xc0001606d8})
/usr/local/go/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1212 +0x218
panic({0xa341c0, 0xc0001606d8})
/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/nspcc-dev/neo-go/pkg/compiler.(*codegen).convertFuncDecl(0xc00015e3c0, {0xc753b8, 0xc000152c80}, 0xc000266300, 0x30)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/codegen.go:497 +0x10b3
github.com/nspcc-dev/neo-go/pkg/compiler.(*codegen).compile.func2(0xc000152c80, 0xc00023c410)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/codegen.go:2153 +0x3f8
github.com/nspcc-dev/neo-go/pkg/compiler.(*codegen).ForEachFile.func1(0xc000229b80)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/compiler.go:102 +0x82
github.com/nspcc-dev/neo-go/pkg/compiler.(*codegen).ForEachPackage(0xc00015e3c0, 0xc000189bb0)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/compiler.go:93 +0xc6
github.com/nspcc-dev/neo-go/pkg/compiler.(*codegen).ForEachFile(0x999a20, 0xc000130d80)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/compiler.go:99 +0x45
github.com/nspcc-dev/neo-go/pkg/compiler.(*codegen).compile(0xc00015e3c0, 0xc0002669f0, 0x1)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/codegen.go:2140 +0x445
github.com/nspcc-dev/neo-go/pkg/compiler.codeGen(0xc0002669f0)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/codegen.go:2191 +0x353
github.com/nspcc-dev/neo-go/pkg/compiler.CompileWithOptions({0xa6f39a, 0x50b6b3}, {0xc6d1a0, 0xc0002421e0}, 0x0)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/compiler.go:218 +0x65
github.com/nspcc-dev/neo-go/pkg/compiler_test.vmAndCompileInterop(0x5648df, {0xa9bf23, 0x94})
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/vm_test.go:75 +0x113
github.com/nspcc-dev/neo-go/pkg/compiler_test.eval(0xc0002421c0, {0xa9bf23, 0x61be8c7}, {0xa68880, 0xc0002421c0})
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/vm_test.go:36 +0x2d
github.com/nspcc-dev/neo-go/pkg/compiler_test.TestUnnamedMethodReceiver(0x4079f9)
/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/compiler/function_call_test.go:400 +0x4f
testing.tRunner(0xc000204b60, 0xbcebb0)
/usr/local/go/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1306 +0x35a
```
The solution is to use the same approach as for unnamed function
parameters handling introduced in #2204. (c *funcScope).newVariable is
able to properly handle "_" receiver.
C# servers with SessionEnabled=false will return iterator IDs and no session
IDs which can be reported as an error immediately because the iterator can't
be traversed.
And test it with the RPC server.
Notice that getters still return int64 instead of *big.Int, that's because
these values are very limited and technically could even fit into an int (but
that seems to be too dangerous to use for long-term compatibility).
See neo-project/neo#2390. Can't see it there? No wonder, that's why we have
this bug for a year and a half. Not critical, we don't care about versions,
but _very_ annoying.
Somewhat similar to invoker, but changing the state (or just creating a
transaction). Transaction creation could've been put into a structure of its
own, but it seems to be less convenient to use this way.
And determine the need for Null dynamically. For some reason the only dynamic
context is Contract.Call. CALLT is not dynamic and neither is a call from
native contract, go figure...
Which is almost like a NeoFS's toStackParameter() on steroids (except it
doesn't mess with noderoles package, it can be casted to int). RPC client's
Invoke* functions expect Parameters, so make it easy to create them.
They were first introduced in a058598ecc and
then carefully moved in 648e0bb242, but it looks
like they were never used by any external code. This code can be useful on the
server, but the server has its own params package to deal with
parameters. Clients usually create Parameters and then get results as
stackitem.Items, so they don't use this code either. So there is zero point in
keeping it.
Which allows to enable/disable the service, change nodes, keys and other
settings. Unfortunately, atomic.Value doesn't allow Store(nil), so we have to
store a pointer there that can point to nil interface.
It can change the committee even if noone voted. Fixes state diff at block
390726 of T5 testnet where there are no transactions, but committee changes
because there were some registrations in previous 21 blocks.
It's not an ideal solution, but at least it solves the problem for
now. Caveats:
* consensus only needs one method, so it's mirrored to Blockchain
* rpcsrv uses core.* definition of the StateRoot (so technically it might as
well not have an internal Ledger), but it uses core already unfortunately
Hiding it behind blockchainer.Blockchain doesn't improve the testing system,
there is no other implementation of it that can fulfil all the needs of the
neotest and at the same time this limits the functions available to tests.
1. It's not good for pkg/core to import anything from pkg/neorpc.
2. The type is closely tied to the state package, even though it's not stored
in the DB
In case there are no returns in the inlined function, jumps point to the
next instruction and can be omitted. This optimization can be extended
to handle other cases, here we just make sure that already existing code
stays the same.
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
It makes a copy of the resulting set, so the lock can be released
earlier. This helps a lot with iterators that keep Seek() unfinished for a
long time,
Change stack items before marshaling them which makes code in result package
much simpler and not requiring interop, iterator and storage dependencies that
clients shouldn't care about.
This also changes SessionBackedByMPT behavior, now instead of waiting for
traverseiterator call it'll rerun the script immediately if a new session is
created.
It doesn't add anything useful to regular Go types and actually native types
are always better to use in the Client. Especially given that this type is
not used by any code outside of the Client itself.
We've got parameters here and usually we name them Raw when they're
represented by json.RawMessage which is not the case here, so make it a bit
more friendly (the type itself is only used in client internals, so rename is
not a huge problem).
It's absolutely irrelevant for the client and request/response packages should
only contain code that is useful on both sides of the conversation. It's OK
for client tests to reuse this code, but the package is used by external
developers and they shouldn't be bothered with it. Nothing changed
functionally here except WSClient simplification. Fixes#2236.
