Commit graph

445 commits

Author SHA1 Message Date
Evgenii Stratonikov
282b55494b consensus: allow to shutdown service 2020-09-18 12:07:01 +03:00
Roman Khimov
120c4d4406 network: don't compress Inventory messages
They're not really compressable.
2020-09-09 20:46:31 +03:00
Roman Khimov
b78bc7f097 network: fix tx requests, we can't ask more than 500 txes at once 2020-09-09 20:46:31 +03:00
Roman Khimov
097b2b8e78 network: fail fast in iteratePeersWithSendMsg
This easily saves us some allocations for single node.
2020-09-09 20:46:31 +03:00
Evgenii Stratonikov
98ca17aab8 network: restrict block queue size
Close #1374.
2020-09-02 17:04:49 +03:00
Evgenii Stratonikov
7e34072519 core: implement (*Blockchain).VerifyWitness
`ScriptFromWitness` is no longer useful, because we support
contract verification.
2020-08-22 12:45:20 +03:00
Roman Khimov
55b2cbb74d core: refactor and improve verification and pooling
Now we have VerifyTx() and PoolTx() APIs that either verify transaction in
isolation or verify it against the mempool (either the primary one or the one
given) and then add it there. There is no possibility to check against the
mempool, but not add a transaction to it, but I doubt we really need it.

It allows to remove some duplication between old PoolTx and verifyTx where
they both tried to check transaction against mempool (verifying first and then
adding it). It also saves us utility token balance check because it's done by
the mempool anyway and we no longer need to do that explicitly in verifyTx.

It makes AddBlock() and verifyBlock() transaction's checks more correct,
because previously they could miss that even though sender S has enough
balance to pay for A, B or C, he can't pay for all of them.

Caveats:
 * consensus is running concurrently to other processes, so things could
   change while verifyBlock() is iterating over transactions, this will be
   mitigated in subsequent commits

Improves TPS value for single node by at least 11%.

Fixes #667, fixes #668.
2020-08-20 18:50:18 +03:00
Anna Shaleva
18691430fd network: decode CMDNotFound
We don't react on this command, but we should be able to decode it.
2020-08-18 14:24:27 +03:00
Roman Khimov
8d19f0e6f5 network: don't request block we already have
GetBlockByIndex handler starts sending blocks right from the start index and
if that index is s.chain.BlockHeight() then we're requesting and receiving a
block we already have.
2020-08-14 16:25:13 +03:00
Roman Khimov
c8cc91eeee network: request blocks when there is a ping with bigger than ours height
Turns out, C# node no longer broadcasts an Inv when it's creating a block,
instead it sends a ping and if we're not paying attention to the height
specified there we're technically missing a new block. Of course we'll get it
later after ping timer expiration and regular ping/pong sequence, but that's
delaying it for no good reason.
2020-08-14 16:22:15 +03:00
Roman Khimov
b1034d8ed6 state: drop Neo 2 Account and everything related
It's substituted with NEP5Balances now.
2020-08-11 20:42:02 +03:00
Roman Khimov
5a42b5c7ae network: correct block addition check
Fixes missing an error on block addition when the header actually went it, but
the block didn't.
2020-08-10 16:51:56 +03:00
Evgenii Stratonikov
807338f97e core: do not store NEP5 transfer log in memory
Traversing transfer log instead of accumulating and returning it
is faster and takes less memory.
2020-08-07 18:21:06 +03:00
Roman Khimov
c3c88a57cd
Merge pull request #1281 from nspcc-dev/drop-go-1.12-and-fix-some-things
Drop go 1.12 and fix some things
2020-08-07 13:34:54 +03:00
Roman Khimov
791c983304 core: drop GetScriptHashesForVerifying
It no longer depends on blockchain state and there can't ever be an error, in
fact we can always iterate over signers, so copying these hashes doesn't make
much sense at all as well as sorting arrays in verifyTxWitnesses (witnesses
order must match signers order).
2020-08-07 12:21:52 +03:00
Roman Khimov
0e2784cd2c always wrap errors when creating new ones with fmt.Errorf()
It doesn't really change anything in most of the cases, but it's a useful
habit anyway.

Fix #350.
2020-08-07 12:21:52 +03:00
Roman Khimov
205f52c563 core: use error wrapping to provide more details 2020-08-07 12:21:52 +03:00
Roman Khimov
5ef08f60ae remove github.com/pkg/errors from dependencies
It's not needed any more with Go 1.13 as we have wrapping/unwrapping in base
packages. All errors.Wrap calls are replaced with fmt.Errorf, some strings are
improved along the way.
2020-08-07 12:21:52 +03:00
Evgenii Stratonikov
27169d140f core: implement (*Blockchain).GetStandByCommitee() 2020-08-06 20:39:13 +03:00
Anna Shaleva
6c8accf18c core, network: request blocks instead of headers
Closes #1192

1. We now have CMDGetBlockByIndex, so there's no need to request headers
   first when we can just ask for blocks.
2. We don't ask for headers (i.e. we don't send CMDGetHeaders),
   consequently, we shouldn't react on CMDHeaders.
3. But we still keep on reacting on CMDGetHeaders command as
   there could be a node which needs headers.
2020-08-04 17:52:34 +03:00
Anna Shaleva
f6f3863e0e network: allow to GetHeaders by index
Use GetBlockByIndex payload for GetHeaders command instead of GetBlocks
payload.
2020-08-04 17:52:34 +03:00
Anna Shaleva
7b1c305000 network: fix handleGetBlockByIndexCmd method
It returned an error in case if block wasn't found (it might be when our
chain is lower). Fixed. It also should return all requested blocks, not
the first one.
2020-08-04 17:52:34 +03:00
Anna Shaleva
0b0591fc34 network: allow to use -1 to specify GetBlockByIndex.Count 2020-08-04 17:52:34 +03:00
Anna Shaleva
0b856033b0 network: use MaxHeadersAllowed to restrict GetBlockByIndex 2020-08-04 17:52:34 +03:00
Anna Shaleva
737ba700e9 network: rename GetBlockData command
GetBlockData -> GetBlockByIndex
2020-08-04 17:52:34 +03:00
Roman Khimov
62f5aa8eb4
Merge pull request #1254 from nspcc-dev/native/policy/maxblocksystemfee
core, consensus: add maxBlockSystemFee to native Policy
2020-08-04 13:13:06 +03:00
Anna Shaleva
db65ed04d9 consensus: apply policy during verifyBlock
To follow C# implementation we should also check proposed block on
policy matching.
2020-08-04 10:59:09 +03:00
Roman Khimov
1f46f73d12 network: handle length mismatch in decompression routine
It's a protocol level error when uncompressed payload size doesn't match the
one specified in the header.
2020-08-03 22:38:55 +03:00
fyrchik
babd84ec10
Merge pull request #1243 from nspcc-dev/protocol/uncompressed_payload_size
network: add uncompressed payload size
2020-07-31 10:03:55 +03:00
fyrchik
e4fc655115
Merge pull request #1239 from nspcc-dev/store_nep5_with_id
core: store contract IDs instead of hashes for NEP5Balances and Transfers
2020-07-30 12:56:19 +03:00
Anna Shaleva
2574b65b74 network: add uncompressed payload size
Closes #1212
2020-07-30 12:47:42 +03:00
Evgenii Stratonikov
0e29382035 core: update MPT during block processing
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2020-07-30 12:42:15 +03:00
Anna Shaleva
dbd460d883 core: retrieve contract hash by ID
We'll need this ability further to retrieve contracts hashes for Nep5Balances.
2020-07-29 15:14:24 +03:00
Roman Khimov
432cef59f4 network: copy peers for Shutdown iteration
We can't lock them (or there will be a deadlock), but we need to fix this:

fatal error: concurrent map iteration and map write

goroutine 1 [running]:
runtime.throw(0xdec086, 0x26)
        /usr/lib64/go/1.12/src/runtime/panic.go:617 +0x72 fp=0xc02fec2bf8 sp=0xc02fec2bc8 pc=0x42d932
runtime.mapiternext(0xc02fec2d40)
        /usr/lib64/go/1.12/src/runtime/map.go:860 +0x597 fp=0xc02fec2c80 sp=0xc02fec2bf8 pc=0x40efe7
github.com/nspcc-dev/neo-go/pkg/network.(*Server).Shutdown(0xc0000fc160)
        /home/rik/dev/neo-go2/pkg/network/server.go:194 +0x238 fp=0xc02fec2db0 sp=0xc02fec2c80 pc=0xa89da8
github.com/nspcc-dev/neo-go/cli/server.startServer(0xc0000fcc60, 0x0, 0x0)
        /home/rik/dev/neo-go2/cli/server/server.go:399 +0x7a9 fp=0xc02fec3820 sp=0xc02fec2db0 pc=0xae2079
...
2020-07-17 19:03:12 +03:00
Anna Shaleva
db5b42b601 network: update CMDUncknown
Closes #1135
2020-07-16 06:58:55 +03:00
Roman Khimov
7eef895061 consensus: use GetNextBlockValidators where appropriate
GetValidators without parameter is called upon DBFT initialization and it
should receive validators for the next block (that will create it),
parameterized GetValidators is used for NextConsensus calculation where we
need a list for the current state of the chain.
2020-07-11 19:54:50 +03:00
Anna Shaleva
abe3c94b95 core: use big.Int to store NEP5 balances
closes #1133
2020-07-09 13:26:39 +03:00
Roman Khimov
08c516014f
Merge pull request #1132 from nspcc-dev/neo3/rpc/fixed8_marshalling
*: switch from Fixed8 to int64
2020-06-29 21:44:40 +03:00
Anna Shaleva
0aaaf7f787 *: switch from fixed8 to int64 in (Blockchain).CalculateClaimable 2020-06-29 21:40:54 +03:00
Anna Shaleva
73b630db9b *: switch from fixed8 to int64
Follow C# implementation, we have to marshall JSON Fixed8 fields without
taking into account decimals.
2020-06-29 21:39:27 +03:00
Roman Khimov
372dd71708
Merge pull request #1108 from nspcc-dev/neo3/compiler/nef
compiler: support *.nef and *.manifest.json generation
2020-06-29 20:55:47 +03:00
Anna Shaleva
6b8957243a *: move wallet config from wallet to config package
In order to avoid dependency cycle at the next commits:

	imports github.com/nspcc-dev/neo-go/pkg/config
	imports github.com/nspcc-dev/neo-go/pkg/wallet
	imports github.com/nspcc-dev/neo-go/pkg/vm
	imports github.com/nspcc-dev/neo-go/pkg/smartcontract/nef
	imports github.com/nspcc-dev/neo-go/pkg/config
2020-06-29 09:15:29 +03:00
Evgenii Stratonikov
857ced8230 consensus: add network magic to hash data
Consensus payload's hash now depends on the network node is operating
in.
2020-06-26 18:41:04 +03:00
Roman Khimov
21cb7d5701
Merge pull request #1078 from nspcc-dev/feature/mempool
network: implement CMDMempool command
2020-06-24 10:49:42 +03:00
Evgenii Stratonikov
57bb2f73de network: implement CMDMempool command
CMDMempool returns hashes of transactions in mempool in chunks.
2020-06-24 10:46:26 +03:00
Roman Khimov
e5f05790d5 core: cache standby validators in the Blockchain
They never change, so it makes no sense parsing the keys over and over
again. It also simplifies the interface a little.
2020-06-24 10:22:17 +03:00
Anna Shaleva
08cc04c3d5 core: add native policy contract
part of #904
2020-06-24 07:35:07 +03:00
Roman Khimov
892812d539 message: fix Verack processsing
Recent 9f42108ef completely broke handshaking.
2020-06-20 00:28:51 +03:00
Roman Khimov
36a65e3847
Merge pull request #1077 from nspcc-dev/neo3/smartcontract/id
core: store smartcontract items by id
2020-06-20 00:12:07 +03:00
Anna Shaleva
f1cdcbc99c core: store smartcontract items by id
closes #1037
2020-06-19 20:48:45 +03:00
Evgenii Stratonikov
505029152f payload: always dereference NullPayload
There is no need in extra indirection level.
2020-06-19 15:02:21 +03:00
Evgenii Stratonikov
9f42108ef2 network: allow NullPayload only for specific commands
CMDFilterClear, CMDGetAddr, CMDMempool all have empty payloads.
2020-06-19 15:01:30 +03:00
Roman Khimov
93b0011988
Merge pull request #1069 from nspcc-dev/addr-locking-and-other-fixes
Addr, locking and other fixes
2020-06-18 23:29:19 +03:00
Evgenii Stratonikov
5354352d63 core: remove transaction priority
There is no such thing as high/low priority transactions, as there are
no free transactions anymore and they are ordered by fees contained
in transaction itself.

Closes #1063.
2020-06-18 22:44:10 +03:00
Roman Khimov
bbe174ee4a network: fix inverted logic of address addition to the pool
Fix #1066.
2020-06-18 21:34:45 +03:00
Roman Khimov
b483c38593 block/transaction: add network magic into the hash
We make it explicit in the appropriate Block/Transaction structures, not via a
singleton as C# node does. I think this approach has a bit more potential and
allows better packages reuse for different purposes.
2020-06-18 12:39:50 +03:00
Roman Khimov
dde0763840 network: fix compression handling functions
Turns out, Neo uses block compression and not a streaming lz4 format. And it
doesn't contain uncompressed payload size which makes guessing it somewhat
suboptimal.
2020-06-18 12:12:43 +03:00
Roman Khimov
8388e509df network: don't allocate memory for messages larger than PayloadMaxSize 2020-06-18 12:12:31 +03:00
Roman Khimov
39fd799992 network: fix Compress flag, it's the first bit (index 0) 2020-06-18 12:12:17 +03:00
Roman Khimov
26f11a52d9 config: move NetMode into its own micropackage
It's going to be used a bit more and pulling whole config just for one type is
a bit wrong.
2020-06-18 12:09:57 +03:00
Anna Shaleva
d5355acfa9 rpc: update RPC server invoke* calls
part of #1036
2020-06-11 19:34:13 +03:00
Anna Shaleva
3418e4f7f4 rpc: update getversion RPC-call
closes #1035
2020-06-10 10:47:11 +03:00
Roman Khimov
63f4f34659 mempool: drop TxWithFee type
All the fees are in transaction, this makes no sense.
2020-06-05 19:21:40 +03:00
Roman Khimov
a986e2a064 *: drop support for old on-chain assets
You no longer can transfer them, so creating/renewing/storing doesn't make
much sense.
2020-06-05 19:21:37 +03:00
Roman Khimov
709146f295 transaction: drop Inputs and Outputs, forget UTXO 2020-06-05 19:20:16 +03:00
Roman Khimov
3d18f09def core: fix CalculateClaimable for NEP5 NEO
It's not stored as Fixed8, so calculations need to be adjusted for that.
2020-06-05 19:20:16 +03:00
Roman Khimov
39dfebccc4 core: no longer treat sysfee as claimable
As it's not on Neo 3, it just gets burned and that's it. Only network fee is
being redistributed to CNs.
2020-06-05 19:20:16 +03:00
Roman Khimov
50ed4c5967 core: don't return an error from CalculateClaimable
As it never returns one.
2020-06-05 19:20:16 +03:00
Roman Khimov
657bb7575e rpc: change getunclaimed to getunclaimedgas
getunclaimed doesn't exist on Neo 3 and getunclaimedgas works for NEP5 GAS.
2020-06-05 19:20:16 +03:00
Roman Khimov
eec462b00c network: fix Inv types for Neo 3
They actually use the same types as for messages. Fixes
2020-05-29T00:06:17.593+0300    WARN    peer disconnected       {"addr": "168.62.167.190:20333", "reason": "handling CMDInv message: invalid inventory type", "peerCount": 3}
2020-05-29 15:02:20 +03:00
Roman Khimov
56b24bf83e capability: prevent panic on wrong capabilities receival
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xad7b95]

goroutine 134 [running]:
github.com/nspcc-dev/neo-go/pkg/network/capability.(*Capability).DecodeBinary(0xc00024a300, 0xc00015c900)
        /home/rik/dev/neo-go/pkg/network/capability/capability.go:69 +0x85
github.com/nspcc-dev/neo-go/pkg/io.(*BinReader).ReadArray(0xc00015c900, 0xd54880, 0xc00018ecd8, 0xc00071fbe0, 0x1, 0x1)
        /home/rik/dev/neo-go/pkg/io/binaryReader.go:141 +0x22f
github.com/nspcc-dev/neo-go/pkg/network/capability.(*Capabilities).DecodeBinary(0xc00018ecd8, 0xc00015c900)
        /home/rik/dev/neo-go/pkg/network/capability/capability.go:17 +0x6d
github.com/nspcc-dev/neo-go/pkg/network/payload.(*AddressAndTime).DecodeBinary(0xc00018ecc0, 0xc00015c900)
        /home/rik/dev/neo-go/pkg/network/payload/address.go:34 +0x59
github.com/nspcc-dev/neo-go/pkg/io.(*BinReader).ReadArray(0xc00015c900, 0xcb9e00, 0xc00024a2a0, 0x0, 0x0, 0x0)
        /home/rik/dev/neo-go/pkg/io/binaryReader.go:141 +0x22f
github.com/nspcc-dev/neo-go/pkg/network/payload.(*AddressList).DecodeBinary(0xc00024a2a0, 0xc00015c900)
        /home/rik/dev/neo-go/pkg/network/payload/address.go:78 +0x54
github.com/nspcc-dev/neo-go/pkg/network.(*Message).decodePayload(0xc000277890, 0xc000292020, 0xc0003da000)
        /home/rik/dev/neo-go/pkg/network/message.go:158 +0x31a
github.com/nspcc-dev/neo-go/pkg/network.(*Message).Decode(0xc000277890, 0xc00071ff48, 0xc00036e090, 0xc000276e40)
        /home/rik/dev/neo-go/pkg/network/message.go:114 +0x19b
github.com/nspcc-dev/neo-go/pkg/network.(*TCPPeer).handleConn(0xc00036e090)
        /home/rik/dev/neo-go/pkg/network/tcp_peer.go:154 +0x244
created by github.com/nspcc-dev/neo-go/pkg/network.(*TCPTransport).Dial
        /home/rik/dev/neo-go/pkg/network/tcp_transport.go:40 +0x1ad
2020-05-29 00:11:54 +03:00
Anna Shaleva
1dffe0bea9 network: remove outdated fields
After #980 there were outdated fields present in message test.
2020-05-27 19:32:50 +03:00
Roman Khimov
a446821753
Merge pull request #980 from nspcc-dev/neo3/protocol/version_payload_optimisation
protocol: add node capabilities
2020-05-27 19:13:00 +03:00
Anna Shaleva
7547a3efe1 network: minor code refactoring
We need to handle IPv6 addresses correctly and net.JoinHostPort takes it
into account.
2020-05-27 19:04:09 +03:00
Anna Shaleva
8c5c248e79 protocol: add capabilities to address payload
Part of #871
2020-05-27 19:02:25 +03:00
Anna Shaleva
c590cc02f4 protocol: add capabilities to version payload
closes #871
2020-05-27 19:01:14 +03:00
Roman Khimov
5733573d2e
Merge pull request #987 from nspcc-dev/neo3/protocol/compression
protocol: add payload compression
2020-05-26 23:31:17 +03:00
Anna Shaleva
0cbd7823ab protocol: add payload compression
closes #872
2020-05-26 22:19:10 +03:00
Roman Khimov
b7d2b659b4 network: get blocks directly from the chain for rebroadcasting
Simplify network<->consensus relations, also broadcast blocks received by
other means like RPC.
2020-05-25 00:27:39 +03:00
Roman Khimov
462022bbdd consensus: remove OnNewBlock(), use Blockchain subscription
Get new blocks directly from the Blockchain. It may lead to some duplications
(as we'll also receive our own blocks), but at the same time it's more
correct, because technically we can also get blocks via other means besides
network server like RPC (submitblock call). And it simplifies network server
at the same time.
2020-05-25 00:27:39 +03:00
Roman Khimov
3ca71d5e8d core: add Blockchain event subscription mechanism
A deep internal part of #895. Blockchainer interface is also extended for
various uses of these methods.
2020-05-25 00:27:39 +03:00
Roman Khimov
bd98940a54
Merge pull request #982 from nspcc-dev/neo3/protocol/getblockdata
protocol: implement getblockdata p2p command
2020-05-22 19:28:42 +03:00
Anna Shaleva
8142caaf8b protocol: implement getblockdata p2p command
closes #891
2020-05-22 19:16:49 +03:00
Anna Shaleva
2f6a3e9af5 protocol: refactor getblocks payload
closes #890
2020-05-22 17:33:59 +03:00
Anna Shaleva
64a2fb63e1 protocol: move magic exchange to version payload
closes #889
2020-05-21 14:23:41 +03:00
Anna Shaleva
3bcc56bdcf protocol: switch to binary MessageCommand
closes #888
2020-05-21 13:57:49 +03:00
Anna Shaleva
1afd630169 core: add senders' fees monitor to mempool
In order to simplify maintainance of overall current senders` fees
in the mempool we are to keep them in a separate map.
2020-05-20 23:34:25 +03:00
Anna Shaleva
7acf5b2841 core: add SystemFee and NetworkFee to transaction
closes #831
2020-05-20 23:26:48 +03:00
Anna Shaleva
29d321b5e1 *: drop miner transaction
1. Completely remove miner transaction

2. Change validation rule for block: block without transactions is
valid.
2020-04-27 17:57:37 +03:00
Anna Shaleva
aa554f0a9a core: update block timestamp format
Changed block.Timestamp from uint32 to uint64
2020-04-27 17:56:44 +03:00
Roman Khimov
b83e84ca08 core: switch to the new voting system (via native NEO contract)
It has all the methods required now, so you can register, vote and get
voting results. Fixes #865.
2020-04-27 16:06:50 +03:00
Evgenii Stratonikov
c508a36827 core,network: shutdown services in tests properly 2020-04-22 17:53:59 +03:00
Roman Khimov
9a67bfac0b network: drop checksums from messages
Follow neo-project/neo#710 changes.
2020-04-19 23:40:31 +03:00
Evgenii Stratonikov
8a2130f5b6 core: extend Blockchainer with GetStandByValidators()
A list of standby validators can be needed in native contracts.
2020-04-16 15:54:58 +03:00
Evgenii Stratonikov
a7c19d445b core: move Blockchainer interface to a separate package 2020-04-11 10:56:36 +03:00
Roman Khimov
5f09381cf4 core: wrap cached dao properly, don't miss cached data
Fixes #817 where invoked contract missed updated account information because
it got it one layer below cachedDao used to process the block.
2020-04-03 10:15:11 +03:00
Roman Khimov
9b5dab57e8
Merge pull request #810 from nspcc-dev/refactor/config
config: move config.go out of config
2020-04-01 10:29:57 +03:00
Anna Shaleva
5a984fdf88 config: move config.go out of config/
closes #423
2020-03-31 17:55:59 +03:00
Evgenii Stratonikov
9abda40171 testserdes: implement helpers for encode/decode routines
Frequently one needs to check if struct serializes/deserializes
properly. This commit implements helpers for such cases including:
1. JSON
2. io.Serializable interface
2020-03-27 10:27:46 +03:00
Evgenii Stratonikov
df2598c8dc core: store NEP5 balances separately
There is no need to take and unmarshal an account structure only
to get it's NEP5 balances.
2020-03-12 12:58:28 +03:00
Roman Khimov
5f1868af28
Merge pull request #743 from nspcc-dev/optimize-some-db-accesses
Optimize some db accesses
2020-03-11 17:33:36 +03:00
Roman Khimov
377fb382aa core: move (un)SpentCoin structs into the state package
As they're all about the state.
2020-03-11 12:22:52 +03:00
Evgenii Stratonikov
b7dee156e2 network: fix a deadlock in DefaultDiscovery
Why a deadlock can occur:
1. (*DefaultDiscovery).run() has a for loop over requestCh channel.
2. (*DefaultDiscovery).RequestRemote() send to this channel while
    holding a mutex.
3. (*DefaultDiscovery).RegisterBadAddr() tries to take mutex for write.
4. Second select-case can't take mutex for read because of (3).
2020-03-10 15:40:23 +03:00
Anna Shaleva
456a2d55fd rpc: implement getvalidators
closes #714
2020-03-07 18:05:40 +03:00
Roman Khimov
fbdc60b731
Merge pull request #723 from nspcc-dev/feature/nep5
core,rpc: implement NEP5-related logic
2020-03-05 18:33:58 +03:00
Evgenii Stratonikov
95a8fa234f rpc: implement getnep5transfers RPC 2020-03-05 18:22:40 +03:00
Roman Khimov
31948ee4a7 network: add a nil check in (*TCPTransport).Close, prevent panic
You have to try to trigger that, but in the event of server shutdown before it
actually started listening Close will easily panic like this:

2020-03-05T11:48:08.660+0300    INFO    node started    {"blockHeight": 6458, "headerHeight": 15605}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x9d5f0a]

goroutine 1 [running]:
github.com/nspcc-dev/neo-go/pkg/network.(*TCPTransport).Close(0xc000fd7050)
        /home/rik/dev/neo-go/pkg/network/tcp_transport.go:78 +0x2a
github.com/nspcc-dev/neo-go/pkg/network.(*Server).Shutdown(0xc000182280)
        /home/rik/dev/neo-go/pkg/network/server.go:190 +0x189
github.com/nspcc-dev/neo-go/cli/server.startServer(0xc000200160, 0x0, 0x0)
        /home/rik/dev/neo-go/cli/server/server.go:367 +0x79e
github.com/urfave/cli.HandleAction(0xb84860, 0xccf240, 0xc000200160, 0xc0001c4500, 0x0)
        /home/rik/dev/neo-go/vendor/github.com/urfave/cli/app.go:490 +0xc8
github.com/urfave/cli.Command.Run(0xc9a160, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0xca407f, 0x10, 0x0, ...)
        /home/rik/dev/neo-go/vendor/github.com/urfave/cli/command.go:210 +0x996
github.com/urfave/cli.(*App).Run(0xc0001d64e0, 0xc0000ca000, 0x3, 0x3, 0x0, 0x0)
        /home/rik/dev/neo-go/vendor/github.com/urfave/cli/app.go:255 +0x6af
main.main()
        /home/rik/dev/neo-go/cli/main.go:25 +0x505
2020-03-05 11:53:26 +03:00
Roman Khimov
e41d434a49 *: move all packages from CityOfZion to nspcc-dev 2020-03-03 17:21:42 +03:00
Evgenii Stratonikov
7095ec6c51 core: implement (*Blockchain).CalculateClaimable
Calculating amount of GAS that can be claimed is required
for getclaimable RPC.
2020-03-02 18:00:00 +03:00
Roman Khimov
252a9f2f31
Merge pull request #690 from nspcc-dev/feature/getapplicationlog
rpc: implement getapplicationlog RPC
2020-03-02 17:41:32 +03:00
Anna Shaleva
ff4384d7ff rpc: implement getapplicationlog RPC
Closes #500
2020-03-02 17:25:27 +03:00
Evgenii Stratonikov
a3dacd3b74 tests: replace t.Fatal with require where possible
This makes tests less verbose and unifies the style
they are written in.
2020-03-02 17:22:27 +03:00
Evgenii Stratonikov
0ef7a76e84 network: batch transactions on broadcast 2020-03-02 10:37:27 +03:00
Roman Khimov
36197056b1
Merge pull request #691 from nspcc-dev/network-conn-overflow-and-shutdown
Network conn overflow and shutdown
2020-02-28 18:16:29 +03:00
Roman Khimov
e213e69a33 network: rework shutdown sequence of Server
Close transport and disconnect peers right in the Shutdown(), so that no new
connections would be accepted and so that all the peers would be disconnected
correctly (avoiding the same deadlock as in e2116e4c3f).
2020-02-28 16:22:04 +03:00
Roman Khimov
77624a8847 network: add Close() to discoverer, shut it down on exit 2020-02-28 16:22:04 +03:00
Roman Khimov
9f7018503a core/transaction: use new transaction.InOut for References
We don't need a map here, use simpler structures.
2020-02-27 12:00:03 +03:00
Roman Khimov
e2116e4c3f network: don't deadlock on connection overflow
(*Peer).Disconnect send an unregister signal to this goroutine, so invoking it
from here is not a good idea, run it asynchronously.
2020-02-24 15:56:49 +03:00
Roman Khimov
723b33e108 network: implement unconnected/bad peers getters
Which allows node to respond to `getpeers` RPC request correctly.
2020-02-21 15:12:03 +03:00
Roman Khimov
d92e193e63 rpc/network: refactor getpeers logic
Deduplicate and simplify code.
2020-02-21 15:12:03 +03:00
Roman Khimov
eb11e5fb11 core: implement basic policying support, fix #370
Implement mempool and consensus block creation policies, almost the same as
SimplePolicy plugin for C# node provides with two caveats:
 * HighPriorityTxType is not configured and hardcoded to ClaimType
 * BlockedAccounts are not supported

Other than that it allows us to run successfuly as testnet CN, previously our
proposals were rejected because we were proposing blocks with oversized
transactions (that are rejected by PoolTx() now).

Mainnet and testnet configuration files are updated accordingly, but privnet
is left as is with no limits.

Configuration is currently attached to the Blockchain and so is the code that
does policying, it may be moved somewhere in the future, but it works for
now.
2020-02-19 12:19:02 +03:00
Roman Khimov
06daeb44f3 core: make IsLowPriority work with pre-calculated fee
Don't recalculate it again and again.
2020-02-19 12:19:02 +03:00
Roman Khimov
37c48b00b4 consensus/network: reinit dbft after block addition
Don't stall on some height if everyone else have moved up already. Fix #673.
2020-02-19 12:19:02 +03:00
Roman Khimov
c5d54e9992 network: introduce (*Server).IsInSync, start consensus in synced state
We define synchronized state as a combination of minimum number of peers and
chain height being not behind of more than 2/3 of these peers.
2020-02-19 12:13:27 +03:00
Roman Khimov
d1a2296939 network: change the disconnect procedure
We can still lock the (*Server).run with dead peers:

Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: goroutine 40 [select, 871 minutes]:
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*TCPPeer).putPacketIntoQueue(0xc030ab5320, 0xc02f251f20, 0xc00af0dcc0, 0x18, 0x40, 0x100000000000000, 0xffffffffffffffff)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_peer.go:82 +0xf4
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*TCPPeer).EnqueueHPPacket(0xc030ab5320, 0xc00af0dcc0, 0x18, 0x40, 0x1367240, 0xc03090ef98)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_peer.go:124 +0x52
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*Server).iteratePeersWithSendMsg(0xc0000ca000, 0xc00af35800, 0xcb2a58, 0x0)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/server.go:720 +0x12a
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*Server).broadcastHPMessage(...)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/server.go:731
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*Server).run(0xc0000ca000)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/server.go:203 +0xee4
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*Server).Start(0xc0000ca000, 0xc000072ba0)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/server.go:173 +0x2ec
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: created by github.com/CityOfZion/neo-go/cli/server.startServer
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/cli/server/server.go:331 +0x476
...
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: goroutine 2199 [chan send, 870 minutes]:
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*TCPPeer).Disconnect.func1()
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_peer.go:366 +0x85
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: sync.(*Once).Do(0xc030ab403c, 0xc02f262788)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/usr/local/go/src/sync/once.go:44 +0xb3
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*TCPPeer).Disconnect(0xc030ab4000, 0xd92440, 0xc000065a00)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_peer.go:365 +0x6d
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*TCPPeer).SendPing.func1()
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_peer.go:394 +0x42
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: created by time.goFunc
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/usr/local/go/src/time/sleep.go:169 +0x44
...
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: goroutine 3448 [chan send, 854 minutes]:
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: github.com/CityOfZion/neo-go/pkg/network.(*TCPPeer).handleConn(0xc01ed203f0)
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_peer.go:143 +0x6c
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: created by github.com/CityOfZion/neo-go/pkg/network.(*TCPTransport).Accept
Feb 13 16:14:50 neo-go-node-2 neo-go[9448]: #011/go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_transport.go:62 +0x44c
...

The problem is that the select in putPacketIntoQueue() only works the way it
was intended to after the `close(p.done)`, but that happens only after
successful unregistration request send. Thus, do disconnects the other way
around, first unblock queueing and exit goroutines, then destroy the
connection (if it wasn't previously destroyed) and only after that signal to
the Server.
2020-02-13 16:24:46 +03:00
Roman Khimov
7ee8f9c5d8 network: fix networking stalls caused by stale peers
We can leak sending goroutines and stall broadcasts because of already gone
peers that happened to be cached by some s.Peers() user (more than 800 of
these can be seen in nodoka log along with (*Server).run blocking on
CMDGetAddr send):

Feb 10 16:35:15 nodoka neo-go[1563]: goroutine 41 [chan send, 3320 minutes]:
Feb 10 16:35:15 nodoka neo-go[1563]: github.com/CityOfZion/neo-go/pkg/network.(*TCPPeer).putPacketIntoQueue(...)
Feb 10 16:35:15 nodoka neo-go[1563]:         /go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_peer.go:81
Feb 10 16:35:15 nodoka neo-go[1563]: github.com/CityOfZion/neo-go/pkg/network.(*TCPPeer).EnqueueHPPacket(0xc0083d57a0, 0xc017206100, 0x18, 0x40, 0x136a240, 0xc018ef9720)
Feb 10 16:35:15 nodoka neo-go[1563]:         /go/src/github.com/CityOfZion/neo-go/pkg/network/tcp_peer.go:119 +0x98
Feb 10 16:35:15 nodoka neo-go[1563]: github.com/CityOfZion/neo-go/pkg/network.(*Server).iteratePeersWithSendMsg(0xc0000ca000, 0xc0001848a0, 0xcb4550, 0x0)
Feb 10 16:35:15 nodoka neo-go[1563]:         /go/src/github.com/CityOfZion/neo-go/pkg/network/server.go:720 +0x12a
Feb 10 16:35:15 nodoka neo-go[1563]: github.com/CityOfZion/neo-go/pkg/network.(*Server).broadcastHPMessage(...)
Feb 10 16:35:15 nodoka neo-go[1563]:         /go/src/github.com/CityOfZion/neo-go/pkg/network/server.go:731
Feb 10 16:35:15 nodoka neo-go[1563]: github.com/CityOfZion/neo-go/pkg/network.(*Server).run(0xc0000ca000)
Feb 10 16:35:15 nodoka neo-go[1563]:         /go/src/github.com/CityOfZion/neo-go/pkg/network/server.go:203 +0xee4
Feb 10 16:35:15 nodoka neo-go[1563]: github.com/CityOfZion/neo-go/pkg/network.(*Server).Start(0xc0000ca000, 0xc000072c60)
Feb 10 16:35:15 nodoka neo-go[1563]:         /go/src/github.com/CityOfZion/neo-go/pkg/network/server.go:173 +0x2ec
Feb 10 16:35:15 nodoka neo-go[1563]: created by github.com/CityOfZion/neo-go/cli/server.startServer
Feb 10 16:35:15 nodoka neo-go[1563]:         /go/src/github.com/CityOfZion/neo-go/cli/server/server.go:331 +0x476
2020-02-10 18:47:52 +03:00
Roman Khimov
c896e2b731 network: fix minor gofmt issues (spotted by goreportcard) 2020-02-08 15:53:08 +03:00
Roman Khimov
b9b77ac1be network: fix block relaying, don't spit out useless errors
We can only add one block of the given height and we have two competing
goroutines to do that --- consensus and block queue. Whomever adds the block
first shouldn't trigger an error in another one.

Fix block relaying for blocks added via the block queue also, previously one
consensus-generated blocks were broadcasted.
2020-02-06 15:41:56 +03:00
Roman Khimov
70b3839fd0 core/mempool: fix AddBlock and tx pooling concurrency issues
Eliminate races between tx checks and adding them to the mempool, ensure the
chain doesn't change while we're working with the new tx. Ensure only one
block addition attempt could be in progress.
2020-02-06 15:41:52 +03:00
Roman Khimov
f0bb886be3 mempool: make all methods pointer methods
Makes no sense copying the Pool around.
2020-02-04 17:36:11 +03:00
Roman Khimov
e5d0125a3f network: do async broadcast when processing P2P TX
It can lead to some goroutine explosion, but supposedly it's better than
stalling other processing and eventually all of these goroutines should finish
their sends. Note that this doesn't change the behavior for RPC-relayed
transactions that are still waiting for the broadcast to finish ensuring
proper transaction distribution before returning the result to the client.
2020-01-30 14:03:52 +03:00
Roman Khimov
0fcbc697ca network: only tell consensus service about new valid transactions
Transactions can be bad and can duplicate each other, consensus service
shouldn't be bothered with that.
2020-01-30 14:03:52 +03:00
Roman Khimov
fdbaac7a30 network: prevent broadcast queue starving, share time with p2p
Blocked broadcast queue of one peer may affect broadcasting capabilities of
the server, so prevent total blocking of it by p2p queue.
2020-01-30 14:03:52 +03:00
Roman Khimov
b2c4587dad network: fix PeerAddr() for not-yet-handshaked case
If we have already got Version message, we don't need the rest of handshake to
complete before being able to properly answer the PeerAddr() requests. Fixes
some duplicate connections between machines.
2020-01-30 14:03:52 +03:00
Roman Khimov
49bd7aada5 network: log peerCount on connection registration
Symmetrical to disconnects, make it easier to look through the logs.
2020-01-30 14:03:52 +03:00
Roman Khimov
32695b4f40 network: don't unregister connected peers
If we drop connection because we're already connected don't unregister this
connected address because it's connected anyway!
2020-01-30 14:03:52 +03:00
Roman Khimov
eb4ec61b8b network: register connected addr in handleVersionCmd()
Prevent useless attempts to connect to this peer if the peer has already made
a connection to us.
2020-01-30 14:03:52 +03:00
Roman Khimov
9eafec0d1d network: introduce peer-to-peer message queue
This one is designed to give more priority to direct nodes communication, that
is that their messaging would have more priority than generic broadcasts. It
should improve consensus process under TX pressure and allow to handle
pings in time (preventing disconnects).
2020-01-30 14:03:52 +03:00
Roman Khimov
72e4eb7172 network: fix wrong NewPing() parameters
They have the opposite order, height first and nonce second. It was done wrong
in 4e6ed902 and never fixed since. Fixes sending wrong peer state leading to
useless getheaders messages (and disconnects when the other side is lagging
behind).
2020-01-30 14:03:52 +03:00
Roman Khimov
da09fda575 network: add debug on message receival 2020-01-30 14:03:52 +03:00
Roman Khimov
1c28dd2567 network: add message type to disconnect error message
If it was caused by message processing, but only after the handshake to
preserve errIdenticalID and other handshaking errors.
2020-01-30 14:03:52 +03:00
Roman Khimov
b79abd1e27 network: detect duplicate connections before handshake completes
We can have more than one connection attempt in progress and not yet completed
the handshake, so if there is a Version already received we should look it.
2020-01-30 14:03:52 +03:00
Roman Khimov
06c3fbe455 network: rework ping sends, fix overpinging
Our node was too pingy because of wrong timer setups (that divided timeout
Duration by time.Second), it also was wrong in its time calculations (using
UTC time to calculate intervals). At the same time missing block is a
server-wide problem, so it's better solved with server-wide protocol loop.
2020-01-28 17:39:52 +03:00
Roman Khimov
39800aecb3 network: fix getdata handling
It was broken by 0ba6b2a7, the peer only responded with one message instead of
full requested set.
2020-01-28 17:39:52 +03:00
Roman Khimov
9aa5d0cf1e
Merge pull request #615 from nspcc-dev/network-peer-updates
Network peer updates
2020-01-23 13:15:43 +03:00
Roman Khimov
99dfdc19e7 network: drop now useless addrReq queue from the server
Just broadcast a high-priority message to everyone.
2020-01-22 11:28:59 +03:00
Roman Khimov
ea3b76ded1 network: make NewServer return an error, fix #612
It can return nil in two cases, so we're better return an error and handle
it.
2020-01-22 11:17:51 +03:00
Roman Khimov
f2ffffddb7 network: rework broadcasting functions, tune priorities
This gives more priority to anything related to consensus.
2020-01-22 11:01:13 +03:00
Roman Khimov
34b863d645 network: introduce Server's MkMsg()
That wraps NewMessage() for a configured network.
2020-01-21 17:31:51 +03:00
Roman Khimov
1f672e0da7 network: move SendVersion() to the Peer
Only leave server-specific `getVersionMsg()` in the Server, all the other
logic is peer-related.
2020-01-21 17:26:08 +03:00