Commit graph

752 commits

Author SHA1 Message Date
Evgenii Stratonikov
351fdd469f core: calcultate NEO owner on-the-fly 2020-04-23 10:36:44 +03:00
Evgenii Stratonikov
1e50016a91 core,test: move helper functions for testchain to the internal package
Our test chain is used in core,rpc and (in future) interop packages.
It is better to have all related declarations in one place to avoid
code duplication.
2020-04-22 18:17:11 +03:00
Evgenii Stratonikov
c508a36827 core,network: shutdown services in tests properly 2020-04-22 17:53:59 +03:00
Anna Shaleva
2b5c14160c core: add sender field to transaction
closes #860
2020-04-20 17:21:28 +03:00
Evgenii Stratonikov
cde4ccf01c vm: remove CHECKSIG/VERIFY/CHECKMULTISIG opcodes 2020-04-20 11:55:24 +03:00
Evgenii Stratonikov
4740d937aa vm: dont use SetCheckedHash outside of vm package 2020-04-20 11:55:24 +03:00
Evgenii Stratonikov
941410a840 core: change verification scripts to new format
Verification scripts now invoke Neo.Crypto.* interops instead of
CHECKSIG/VERIFY opcodes.
2020-04-20 11:55:24 +03:00
Roman Khimov
ab8296bc57
Merge pull request #857 from nspcc-dev/neo3/verifiable
core,crypto: implement Verifiable interface
2020-04-17 13:06:55 +03:00
Evgenii Stratonikov
ceff8736f2 vm: use ID-based syscalls
In NEO3 SYSCALL opcode has 4-byte ID parameter.
This commit removes support for string-based syscalls and
changes SYSCALL's parameter to be fixed 4-byte value.
2020-04-17 11:46:31 +03:00
Evgenii Stratonikov
bfbbef952a vm: move InteropNameToID to emit package 2020-04-17 11:46:31 +03:00
Evgenii Stratonikov
82b230f19f core: rename *block.Base.GetHashableData to GetSignedPart()
This allow to use `Block` as a Verifiable item.
When tx is provided, it is set as an interop's script container.
Otherwise, block is set.
2020-04-17 11:12:40 +03:00
Evgenii Stratonikov
a92872931c interop/crypto: allow ECDsaVerify to verify ScriptContainer
When verifying transaction or block, verification script can be
a simple PUSHNULL + SYSCALL, which means that script-encontaining
entity should be verified.
2020-04-17 11:12:40 +03:00
Evgenii Stratonikov
8f08065a8e interop/crypto: allow ECDsaVerify to accept interop items
When invokes with interop item on stack, it should check
for the signature of Verifiable item it contains.
2020-04-17 11:12:40 +03:00
Roman Khimov
888be75445
Merge pull request #863 from nspcc-dev/drop-publish-tx
Drop publish tx
2020-04-16 21:23:37 +03:00
Evgenii Stratonikov
76700f31cf core: implement skeletons for NEO/GAS native contracts 2020-04-16 15:55:35 +03:00
Evgenii Stratonikov
25354c44f9 core: implement NativeContract support 2020-04-16 15:55:34 +03:00
Evgenii Stratonikov
9586af32f2 core: move runtime.CheckWitness interop to a separate package 2020-04-16 15:54:58 +03:00
Evgenii Stratonikov
8a2130f5b6 core: extend Blockchainer with GetStandByValidators()
A list of standby validators can be needed in native contracts.
2020-04-16 15:54:58 +03:00
Evgenii Stratonikov
b446753c57 core,vm: move get/putContextScriptHash to vm package 2020-04-16 15:54:52 +03:00
Roman Khimov
40991d3be3 transaction: drop AgencyType and VotingType
I have no idea what are they about.
2020-04-16 14:44:46 +03:00
Roman Khimov
4912e4f425 *: drop Publish transaction type
It shouldn't be used even in NEO 2 as it was substituted by contract
deployment interop functions.
2020-04-16 14:40:20 +03:00
Anna Shaleva
5fa11987d2 core: add validUntilBlock field to transaction
1. closes #841

2. Commented out test cases where binary transaction are used.
These test cases marked with `TODO NEO3.0: Update binary` and need to be
updated.

3. Updated other tests.

4. Added cache to calculateValidUntilBlock() RPC-client method.
2020-04-15 13:46:43 +03:00
Roman Khimov
1e3c36433f
Merge pull request #844 from nspcc-dev/neo3/nonce
core: add nonce field to transaction
2020-04-14 16:24:02 +03:00
Anna Shaleva
65503aa9b4 core: add nonce field to transaction
1. Closes #840: added Nonce field to transaction.Transaction and
removed Nonce field from transaction.MinerTx

2. Added following methods to different tx types:
  - NewMinerTx()
  - NewMinerTxWithNonce(...)
  - NewEnrollmentTx(...)
  - NewIssueTx()
  - NewPublishTx(...)
  - NewRegisterTx(...)
  - NewStateTx(...)
in order to avoid code duplication when new transaction is created.

3. Commented out test cases where binary transaction/block are used.
These test cases marked with `TODO NEO3.0: Update binary` and need to be
updated.

4. Updated other tests

5. Added constant Nonce to GoveringTockenTx, UtilityTokenTx and genesis
block to avoid data variability. Also marked with TODO.
2020-04-14 16:19:41 +03:00
Evgenii Stratonikov
a99e0d7ba6 core: fix typo in TestECDSAVerify 2020-04-14 16:15:50 +03:00
Roman Khimov
ace5614a3d
Merge pull request #783 from nspcc-dev/neo3/movecrypto
Implement Neo.Crypto.* interops
2020-04-14 15:54:30 +03:00
Roman Khimov
c40d7aa7f4 core: bump DB version to make it differ from 2.0 branch 2020-04-14 12:54:57 +03:00
Evgenii Stratonikov
c23af595c9 core: implement Neo.Crypto.ECDsaCheckMultisig interop 2020-04-13 13:47:39 +03:00
Evgenii Stratonikov
5a30af2c75 core: implement Neo.Crypto.ECDsaVerify interop 2020-04-13 13:46:31 +03:00
Evgenii Stratonikov
1611ede58c crypto/keys: implement NewPublicKeyFromBytes()
It is convenient to have a single function instead of
allocating new `PublicKey` and using `DecodeBytes()` on it.
2020-04-13 13:13:15 +03:00
Anna Shaleva
d1f92a585b core: add NewTransactionFromBytes method to Transaction
Added `NewTransactionFromBytes(b []byte)` method to
transaction.Transaction in order to avoid code duplication.
2020-04-13 12:04:43 +03:00
Evgenii Stratonikov
30cc2c5783 core: move enumerator/iterator interops to a separate packages 2020-04-11 10:56:36 +03:00
Evgenii Stratonikov
7ffc6c0936 core: move interopContext to a separate package 2020-04-11 10:56:36 +03:00
Evgenii Stratonikov
90a08986d6 core: fix typo in function doc-comment 2020-04-11 10:56:36 +03:00
Evgenii Stratonikov
89d1f7ce75 core: remove unneeded comment
Serialization interops are already implemented.
2020-04-11 10:56:36 +03:00
Evgenii Stratonikov
a7c19d445b core: move Blockchainer interface to a separate package 2020-04-11 10:56:36 +03:00
Evgenii Stratonikov
efddcf3bfe core: remove interop methods from interopContext
If interops are defined as a separate functions
they can be implemented in a separate packages
which can help us to structure core.
2020-04-11 10:56:35 +03:00
Anna Shaleva
54cdfe4a23 storage: add support of BadgerDB
closes #820
2020-04-09 13:55:59 +03:00
Evgenii Stratonikov
ee0ba9b1b4 core: make SpawnVM a method of context
spawnVMWithInterops is rather long too type and
it doesn't use Blockchain in any way.
2020-04-08 08:38:45 +03:00
Evgenii Stratonikov
a71cd0961e core/dao: remove unnecessary slice type
It is used only once, so a simple `sort.Slice`
invocation will suffice.
2020-04-08 08:38:45 +03:00
Evgenii Stratonikov
030b7754ad core: move DAO to a separate package 2020-04-08 08:38:44 +03:00
Roman Khimov
9997661998
Merge pull request #821 from nspcc-dev/fix-cache-propagation-to-invocations
core: wrap cached dao properly, don't miss cached data
2020-04-06 11:52:41 +03:00
Evgenii Stratonikov
1fcc019bf3 rpc: update test chain
Also provide info for getblockheader RPC while
generating test chain.
2020-04-06 09:31:09 +03:00
Roman Khimov
5f09381cf4 core: wrap cached dao properly, don't miss cached data
Fixes #817 where invoked contract missed updated account information because
it got it one layer below cachedDao used to process the block.
2020-04-03 10:15:11 +03:00
Roman Khimov
2d0ad30fcf vm: rework Map with internal slice representation
Which makes iterating over map stable which is important for serialization and
and even fixes occasional test failures. We use the same ordering here as
NEO 3.0 uses, but it should also be fine for NEO 2.0 because it has no
defined order.
2020-04-01 19:33:53 +03:00
Roman Khimov
9b5dab57e8
Merge pull request #810 from nspcc-dev/refactor/config
config: move config.go out of config
2020-04-01 10:29:57 +03:00
Anna Shaleva
5a984fdf88 config: move config.go out of config/
closes #423
2020-03-31 17:55:59 +03:00
Roman Khimov
e41853d0a4
Merge pull request #789 from nspcc-dev/fix/tx_specific_data_marshalling
rpc: fix marshalling of type-specific tx data
2020-03-30 16:10:25 +03:00
Anna Shaleva
9c09ad9c89 rpc: fix marshalling of type-specific tx data
closes #585
2020-03-30 15:48:50 +03:00
Roman Khimov
237de73dec
Merge pull request #807 from nspcc-dev/optimize-memcached-persist
storage: optimize (*MemCachedStore).Persist for memory-backed ps
2020-03-29 13:41:21 +03:00
Roman Khimov
4758de71ec storage: optimize (*MemCachedStore).Persist for memory-backed ps
Most of the time it's persisted into the MemoryStore or MemCachedStore, when
that's the case there is no real need to go through the Batch mechanism as it
incurs multiple copies of the data.

Importing 1.5M mainnet blocks with verification turned off, before:
real    12m39,484s
user    20m48,300s
sys     2m25,022s

After:
real    11m15,053s
user    18m2,755s
sys     2m4,162s

So it's around 10% improvement which looks good enough.
2020-03-28 17:21:50 +03:00
Roman Khimov
c738975b7b
Merge pull request #803 from nspcc-dev/feature/appcall
emit: implement AppCallWithOperationAndArgs
2020-03-27 13:30:48 +03:00
Evgenii Stratonikov
db2dccf7cb emit: implement AppCallWithOperationAndArgs
It is nice to have a typical task of calling contract method
with specific arguments incapsulated inside some function.
2020-03-27 11:05:36 +03:00
Evgenii Stratonikov
0036b3e52b random: make use or random package in tests
Also implement Bytes/Fill routines for generating byte slices.
2020-03-27 10:27:46 +03:00
Evgenii Stratonikov
9abda40171 testserdes: implement helpers for encode/decode routines
Frequently one needs to check if struct serializes/deserializes
properly. This commit implements helpers for such cases including:
1. JSON
2. io.Serializable interface
2020-03-27 10:27:46 +03:00
Evgenii Stratonikov
fcc4877f43 go.mod: update boltdb to v1.3.4
Also import lib as declared in it's go.mod:
via `go.etcd.io/bbolt`.
2020-03-25 17:07:54 +03:00
Evgenii Stratonikov
c992d6c518 core: reuse buffer in (*cacheddao).Persist()
When serializing multiple accounts, cost of a buffer grow
can become significant. This commit tries to amortize it by
reusing the same buffer in a single `Persist()` call.
2020-03-25 13:13:54 +03:00
Evgenii Stratonikov
6ed2bd63b9 core: marshal AppExecResult.Stack as an array
Closes #782.
2020-03-23 16:27:47 +03:00
Roman Khimov
8400f0add2 mempool: gofmt -s 2020-03-17 16:47:51 +03:00
Roman Khimov
ec76a0bf15 mempool: disallow more than one issue tx at once
Technically they could conflict for available asset amount, but as they're
very rare (and even can be considered obsolete) we can simplify this check.
2020-03-17 11:40:01 +03:00
Roman Khimov
d5d0479671 core: verify results of issue transaction
It shouldn't try to issue more tokens than there is available.
2020-03-17 11:40:01 +03:00
Roman Khimov
10601cb375 core: add issuer hashes into the verification list for Issue TX
As it should be done.
2020-03-17 11:40:00 +03:00
Roman Khimov
dca637d2d0 core: add owner hash to verified list for Register TX
As it should be done.
2020-03-17 11:37:02 +03:00
Roman Khimov
77a799f7d0 core: add missing state tx verifications 2020-03-16 20:14:59 +03:00
Roman Khimov
6ede65610d core: add appropriate hashes to check for State TX
These checks are important for proper transaction verification.
2020-03-16 19:52:28 +03:00
Evgenii Stratonikov
b193e78def core: remove duplication from IsDoubleClaim/IsDoubleSpend 2020-03-16 14:11:19 +03:00
Evgenii Stratonikov
cdf025bf89 transaction: implement AddVerificationHash() method 2020-03-16 14:11:19 +03:00
Roman Khimov
e9429374aa
Merge pull request #756 from nspcc-dev/feature/unclaimed
core/state: do not unmarshal Unclaimed balances in account
2020-03-16 13:21:23 +03:00
Evgenii Stratonikov
e503d1001d core/state: do not unmarshal Unclaimed balances in account 2020-03-16 12:19:32 +03:00
Roman Khimov
e6e8761d35 core: fix contract's state migration and don't swallow errors
Fixes difference in state changes at mainnet's block 2442790 because contract
migration in b4eb2dc35226e6520ee4e09a56197dff91547b50a7f57edc82930fc18c75dffc
doesn't actually transfer the storage state, it only deletes the old one.

And add an error check just in case.
2020-03-16 11:52:09 +03:00
Roman Khimov
d9a83373ed
Merge pull request #748 from nspcc-dev/feature/splitnep5
core: store NEP5 balances separately from account
2020-03-12 18:31:25 +03:00
Evgenii Stratonikov
ac475940a0 core: cache NEP5Transfers in cached DAO 2020-03-12 17:16:11 +03:00
Evgenii Stratonikov
6fa2a998f4 core: cache NEP5Balances in cached DAO 2020-03-12 17:06:20 +03:00
Evgenii Stratonikov
32401a567e core: store NEP5Transfers in batches
This is an append-only log which is read only during some RPCs.
It is rather slow to get it from base every time we need to append to
it. This commit stores all NEP5Transfers in batches, so that
only a last batch needs to be unmarshaled during block processing.
2020-03-12 14:51:12 +03:00
Evgenii Stratonikov
3c6d9653b0 core/state: add Size() method to NEP5TransferLog 2020-03-12 12:58:30 +03:00
Evgenii Stratonikov
df2598c8dc core: store NEP5 balances separately
There is no need to take and unmarshal an account structure only
to get it's NEP5 balances.
2020-03-12 12:58:28 +03:00
Roman Khimov
bbd802681e cli: make gas parameter to deployment add gas to the base price
That's how it was intended to behave originally. One thing questionable here
is contract price (policy thing, basically) being moved to smartcontract
package, but it's probably fine for NEO 2.0 (as it won't change) and we'll
make something better for NEO 3.0.
2020-03-11 20:34:36 +03:00
Roman Khimov
5f1868af28
Merge pull request #743 from nspcc-dev/optimize-some-db-accesses
Optimize some db accesses
2020-03-11 17:33:36 +03:00
Roman Khimov
abd7855890
Merge pull request #742 from nspcc-dev/add-dynamic-appcall
vm: add support for dynamic invocations in APPCALL
2020-03-11 17:27:05 +03:00
Roman Khimov
8c902a7223 core: cache UnspentCoins in cachedDao
1.5M block import time (VerifyBlocks disabled) on AMD Ryzen 5 1600/16GB/HDD,
before:
real    159m16.551s
user    69m58.279s
sys     7m34.334s

after:
real    139m41.836s
user    67m12.477s
sys     6m19.420s

12% which is even a bit more than could be expected from inputs analysis (that
has around 10% cache hits for a block-wide cache), worth doing.
2020-03-11 12:40:02 +03:00
Roman Khimov
23464401bc core/state: merge spent and unspent coins state, use it to store more things
This change reduces pressure on DB by doing the following things:
 * not storing additional KV pair for SpentCoin
 * storing Output right in the UnspentCoin, thus eliminating the need to get a
   full transaction from DB

At the same time it makes UnspentCoin more fat and hot, but it should probably
worth it.

Also drop `GetUnspentCoinStateOrNew` as it shouldn't ever existed, UTXOs
can't come out of nowhere.

1.5M block import time (VerifyBlocks disabled) on AMD Ryzen 5 1600/16GB/HDD,
before:
real    302m9.895s
user    96m17.200s
sys     13m37.084s

after:
real    159m16.551s
user    69m58.279s
sys     7m34.334s

So it's almost two-fold which is a great improvement.
2020-03-11 12:40:02 +03:00
Roman Khimov
e1f194ea7b core: treat state.Coin as a bitfield
As it was intended to.
2020-03-11 12:22:57 +03:00
Roman Khimov
377fb382aa core: move (un)SpentCoin structs into the state package
As they're all about the state.
2020-03-11 12:22:52 +03:00
Roman Khimov
8318adac56 vm: add support for dynamic invocations in APPCALL
Fixes #740.
2020-03-10 17:17:36 +03:00
Evgenii Stratonikov
ebf867b11d state: add a test for NEP5Transfer size
It's size is used in NEP5TransferLog so we need to be
sure it reflects reality.
2020-03-10 13:02:14 +03:00
Roman Khimov
ee57b96566
Merge pull request #733 from nspcc-dev/fix-getreferences-interop-regression
Fix GetReferences interop regression
2020-03-10 10:27:36 +03:00
Roman Khimov
0e2bda4f21 core: drop txHash from SpentCoinState
It's a key for it, makes no sense storing it as data.
2020-03-09 16:58:21 +03:00
Roman Khimov
eb404ceae3 core: fix max contract description limit check
It differs from other parameters in C# code. Fixes #735.
2020-03-09 14:18:51 +03:00
Roman Khimov
4587121c7f core: fix error propagation in contract-related interops
Obvious bug that hides failed contract deployments.
2020-03-09 14:18:00 +03:00
Roman Khimov
4b83e9a5cd
Merge pull request #732 from nspcc-dev/feature/getvalidators
rpc: implement getvalidators
2020-03-07 21:43:38 +03:00
Anna Shaleva
456a2d55fd rpc: implement getvalidators
closes #714
2020-03-07 18:05:40 +03:00
Roman Khimov
baeaa3dbe6 core: optimize tx verification, only get references and results once
Getting references requires DB access and that is expensive.
2020-03-06 20:15:01 +03:00
Roman Khimov
fe4916f691 core: simplify GetTransactionResults() a bit
Make less movements.
2020-03-06 19:26:39 +03:00
Roman Khimov
ced5ddbb9e core: fix wrong references ordering in interop function
Broken by 9f7018503a. Almost the same problem as
in 01082a8988 (though it is deterministic now,
just not the way the contract expects).
2020-03-06 19:20:55 +03:00
Roman Khimov
0e8ff558d1
Merge pull request #727 from nspcc-dev/fix-wrong-endian-in-interop-hashes
core: fix wrong endian used in interop functions
2020-03-06 19:07:28 +03:00
Roman Khimov
f8eee778f4
Merge pull request #724 from nspcc-dev/feature/submitblock
rpc: implement submitblock
2020-03-06 12:08:45 +03:00
Anna Shaleva
a746d8e6e6 rpc: implement submitblock RPC
closes #344
2020-03-06 12:03:08 +03:00
Evgenii Stratonikov
40188c5400 core: get rid of unnecessary copies
There is no need to keep Balances and Unclaimed slices
sorted, we need only to remove a single element.
2020-03-05 20:42:20 +03:00
Anna Shaleva
d3063c26e1 core: add custom error to blockhain.go
Add InvalidBlockIndex error to AddBlock func
2020-03-05 20:30:19 +03:00
Roman Khimov
a9abd3d841 core: fix wrong endian used in interop functions
C# uses ToArray() or UintXXX(bytes) here which interprets hashes as they
should be interpreted (BE, although they always convert to LE when converting
to String just for the fun of it). It leads to state difference for us at
block 2025204 where even though we have the same value for the key, the key
itself differs, ours:

dd2b538e2a0c1db1ae5061c15be14f916bd1e678e512ffcda6d9499d8e7fe97ee71fd6b8004583d9afe09cc4dadbd5deb63d01e061009b7cffdaa674beae0f930ebe6085af900093e5fe56b34a5c220ccdcf6efc336fc5000000000000000000000000000000000010

theirs:

dd2b538e2a0c1db1ae5061c15be14f916bd1e67861e0013db6ded5dbdac49ce0afd9834500b8d61fe77ee97f8e9d49d9a6cdff12e5009b7cffdaa674beae0f930ebe6085af900093e5fe56b34a5c220ccdcf6efc336fc5000000000000000000000000000000000010

In this key there is a tx hash encoded
(e512ffcda6d9499d8e7fe97ee71fd6b84583d9afe09cc4dadbd5deb63d01e061 in LE used
by all the tools like neoscan).

I love Neo.
2020-03-05 19:44:09 +03:00
Evgenii Stratonikov
95a8fa234f rpc: implement getnep5transfers RPC 2020-03-05 18:22:40 +03:00
Evgenii Stratonikov
6d270c4550 core,rpc: add NEP5 contract to testdata
Also transfer tokens between accounts.
2020-03-05 18:22:40 +03:00
Evgenii Stratonikov
f92fd3c948 core: track NEP5 transfers 2020-03-05 18:22:40 +03:00
Evgenii Stratonikov
e8c4179a9c core: track NEP5 balances 2020-03-05 18:22:19 +03:00
Evgenii Stratonikov
3a510b9dad core: allow transfer amount to be bytes
VM can produce both big.Int and []byte because they
are converted to each other on demand.
2020-03-05 15:18:38 +03:00
Roman Khimov
d03b2ef4a1
Merge pull request #706 from nspcc-dev/feature/transfer
cli: implement transfer from multisig accounts
2020-03-05 12:28:01 +03:00
Roman Khimov
5c781be08d
Merge pull request #718 from nspcc-dev/fix-invocation-tx-checks-and-fee
Fix Invocation TX checks and fee
2020-03-05 11:51:51 +03:00
Roman Khimov
9ac8001c7d core: protect (*Blockchain).keyCache with a lock
Avoid failures like this:
fatal error: concurrent map writes

goroutine 103 [running]:
runtime.throw(0xca6fe3, 0x15)
        /usr/lib64/go/1.12/src/runtime/panic.go:617 +0x72 fp=0xc000687988 sp=0xc000687958 pc=0x42d8d2
runtime.mapassign(0xb9d4a0, 0xc0000b1f80, 0xc000687a6c, 0x13bd940)
        /usr/lib64/go/1.12/src/runtime/map.go:590 +0x5e3 fp=0xc000687a10 sp=0xc000687988 pc=0x40e2a3
github.com/nspcc-dev/neo-go/pkg/core.(*Blockchain).verifyHashAgainstScript(0xc0000f8000, 0x1300d1f5a3d348be, 0x78607089e4fe9fab, 0xc0a21e4f71, 0xc000d16070, 0x1e2b8881f8178e92, 0xd589cfa965287bb8, 0x24a4e4f292ebc797, 0xb90d3120d878e619, 0xc0001d01e0, ...)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:1866 +0x371 fp=0xc000687ad8 sp=0xc000687a10 pc=0x984ed1
github.com/nspcc-dev/neo-go/pkg/core.(*Blockchain).verifyHeaderWitnesses(0xc0000f8000, 0xc000d16000, 0xc00042d950, 0x1, 0xc00008a6c0)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:1914 +0x209 fp=0xc000687c90 sp=0xc000687ad8 pc=0x985999
github.com/nspcc-dev/neo-go/pkg/core.(*Blockchain).verifyHeader(0xc0000f8000, 0xc000d16000, 0xc00042d950, 0xf0ee0049d611101f, 0xf8967398a8f9351b)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:1254 +0x235 fp=0xc000687d70 sp=0xc000687c90 pc=0x97e4e5
github.com/nspcc-dev/neo-go/pkg/core.(*Blockchain).addHeaders(0xc0000f8000, 0x1, 0xc000d12000, 0x7d0, 0x7d0, 0x0, 0x1)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:369 +0x266 fp=0xc000687e68 sp=0xc000687d70 pc=0x973f46
github.com/nspcc-dev/neo-go/pkg/core.(*Blockchain).AddHeaders(0xc0000f8000, 0xc000d12000, 0x7d0, 0x7d0, 0x0, 0xc0003640c0)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:338 +0x51 fp=0xc000687eb0 sp=0xc000687e68 pc=0x973cb1
github.com/nspcc-dev/neo-go/pkg/network.(*Server).handleHeadersCmd(0xc00022c140, 0xdd0ca0, 0xc000374000, 0xc00000c020)
        /home/rik/dev/neo-go/pkg/network/server.go:431 +0x74 fp=0xc000687fc0 sp=0xc000687eb0 pc=0x9ce5e4
runtime.goexit()
        /usr/lib64/go/1.12/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000687fc8 sp=0xc000687fc0 pc=0x45ca51
created by github.com/nspcc-dev/neo-go/pkg/network.(*Server).handleMessage
        /home/rik/dev/neo-go/pkg/network/server.go:705 +0xa4a
...
2020-03-04 20:26:18 +03:00
Evgenii Stratonikov
7eaeb18f18 transaction: marshal ContractTX even if Data is nil
It contains no information so it doesn't matter anyway.
2020-03-04 20:01:26 +03:00
Evgenii Stratonikov
634e9483d3 transaction: unmarshal Witness properly
Both verification and invocation scripts need to
be unmarshaled from hex.
Also fix failing RPC tests: block contains non-pointer
`transaction.Witness` field and (*Witness).MarshalJSON method
is not called.
2020-03-04 20:01:26 +03:00
Evgenii Stratonikov
33f99104e8 transaction: unmarshal Output properly
Address is marshaled in base58 and needs to be
decoded accordingly.
2020-03-04 20:01:26 +03:00
Roman Khimov
81d89cd502 core: fix SystemFee calculation for Invocation TXes
They have it specified right in the transaction. Unfortunately, this little
change rendered invalid our RPC test chain, but I think it became even better
after it, especially given that chain generation is a nice test by itself, so
it should be running as a regular test.
2020-03-04 19:23:23 +03:00
Roman Khimov
ae6edf0601 core: add a check for fractional gas in invocation TXes
It's not allowed in C# code and that's reasonable as it's a sysfee essentially
that can only be integer.
2020-03-04 19:23:23 +03:00
Roman Khimov
823798514a util: rename Int64Value to IntegralValue, add FractionalValue
Makes it's interface more clear and allows to easily access fractional part.
2020-03-04 19:23:23 +03:00
Roman Khimov
7cc847d655 transaction: add some missing InvocationTX decoding checks
Script must have something inside and GAS can't be negative.
2020-03-04 19:23:23 +03:00
Roman Khimov
e41d434a49 *: move all packages from CityOfZion to nspcc-dev 2020-03-03 17:21:42 +03:00
Roman Khimov
59e6dac351
Merge pull request #708 from nspcc-dev/fix/trigger
core: convert trigger.Type to byte when pushing on stack
2020-03-03 16:17:01 +03:00
Evgenii Stratonikov
fe3bbbd78a core: convert trigger.Type to byte when pushing on stack 2020-03-03 16:13:19 +03:00
Roman Khimov
c3e73c5b7d core: don't reverify stale headers in addHeader
During networked synchronization we expect there to be a lot of duplicate
headers received and it makes no sense for us reverifying them.
2020-03-03 15:34:03 +03:00
Evgenii Stratonikov
05a3625b7d wallet: implement (*Account).SignTx
It is used in both CLI and RPC.
2020-03-02 18:03:56 +03:00
Evgenii Stratonikov
6add4f3e50 transaction: disallow negative outputs
Otherwise it is possible to make outputs which will sum
to the expected value, but steal GAS from some other account.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
9e9d59b49a core: set NetworkFee to 0 for Claim and Miner transactions
Claim tx have no GAS inputs and a positive output which
can lead to negative network fee.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
de5215a564 core: use GetAccountStateOrNew() in (*Blockchain).GetValidators()
Target of the transaction output may not yet exist in database.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
1b9968df67 core: optimize CalculateClaimable()
Because accumulated system fee is stored for every block,
it is easy to calculate sum with just to reads.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
258d8be1dd core: store all accumulated SystemFee with every block 2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
f69c8a763b core: store system fee together with block
Recalculating system fee can be rather costly if done
frequently.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
972e0d8ad1 core: add one more Contract tx to the test chain
When testing CLI it is useful to have some spent coins
on an account with a known key.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
45b8669b42 core: verify Claim transactions 2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
44792ed5f7 core: make test chain format compatible with mainnet
Write uint32 length before every block.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
0e2a1f40ba core: add ContractTX to a testchain generator
Simple transfer from multisig account to the account
of one of the validators.
2020-03-02 18:00:00 +03:00
Evgenii Stratonikov
72d72296c3 core: implement (*Transaction).GetSignedPart()
Marshalling it and taking all but last byte violates incapsulation
and is just wrong in case transaction already contains any witnesses.
2020-03-02 18:00:00 +03:00
Evgenii Stratonikov
95d9f36c98 core: implement UnclaimedBalance tracking
To make it easy to get unclaimed coins for the
specified account they must be tracked together.
2020-03-02 18:00:00 +03:00
Evgenii Stratonikov
7095ec6c51 core: implement (*Blockchain).CalculateClaimable
Calculating amount of GAS that can be claimed is required
for getclaimable RPC.
2020-03-02 18:00:00 +03:00
Roman Khimov
252a9f2f31
Merge pull request #690 from nspcc-dev/feature/getapplicationlog
rpc: implement getapplicationlog RPC
2020-03-02 17:41:32 +03:00
Anna Shaleva
ff4384d7ff rpc: implement getapplicationlog RPC
Closes #500
2020-03-02 17:25:27 +03:00
Anna Shaleva
7d46404e2d smartcontract: turn trigger types into Type
1) Turn trigger types from byte constants into Type
2) Add auto-generated stringer for future purposes
2020-03-02 17:25:27 +03:00
Evgenii Stratonikov
a3dacd3b74 tests: replace t.Fatal with require where possible
This makes tests less verbose and unifies the style
they are written in.
2020-03-02 17:22:27 +03:00
Evgenii Stratonikov
66f96e3f32 core: shutdown Blockchain gracefully in tests 2020-03-02 17:22:27 +03:00
Evgenii Stratonikov
3a5224344e core: verify headers in AddHeaders()
Headers can be malformed so public methods should verify them
before adding.
2020-03-02 17:22:26 +03:00
Evgenii Stratonikov
357bb4ce41 core: get rid of global variables in tests
It can lead to unnecessary race conditions and is just
a bad practice.
2020-03-02 17:04:08 +03:00
Evgenii Stratonikov
8ca94e23c8 core: replace makeBlocks() with addBlocks() in tests
This simplifies tests a bit.
2020-03-02 17:04:08 +03:00
Evgenii Stratonikov
cbf26bac83 mempool: do not allocate new slice for verified transactions
Because transactions a iterated in an increasing order,
we can filter slice in-place.
2020-03-02 10:38:27 +03:00
Roman Khimov
3e2b490025 mempool: rework inputs verification, check Claim txes
Use more efficient check for Input and also check Claims to avoid double
claiming.
2020-02-27 13:43:35 +03:00
Roman Khimov
f329de73e8 mempool: fix Remove bug (dropping wrong tx!)
Uncovered by TestMemPoolAddRemoveWithInputsAndClaims (to be added later)
2020-02-27 13:43:35 +03:00
Roman Khimov
7d59fa0066
Merge pull request #696 from nspcc-dev/tx-verification-fixes
Claim and enrollment TX verification fixes
2020-02-27 12:45:36 +03:00
Roman Khimov
00d199ed85 core: fix NetworkFee calculation
Add doesn't change the variable state. Thanks to GolangCI for catching this.
2020-02-27 12:32:56 +03:00
Roman Khimov
7bea6e043e core: add double claim verification check 2020-02-27 12:32:56 +03:00
Roman Khimov
29586f2aa7 core: add duplicate Claim verification check
Refactor HaveInputsDuplicate() out of the core and Blockchain, it doesn't
depend on the Blockchain state. Make it more efficient.
2020-02-27 12:32:56 +03:00
Roman Khimov
9b1ad40273 core: drop GetScriptHashesForVerifyingClaim, fix Claim's ScriptHash getter
Claim transactions should _add_ claims scripthashes to the standard list, not
replace them. And this code is actually very reusable.
2020-02-27 12:32:56 +03:00
Roman Khimov
9f7018503a core/transaction: use new transaction.InOut for References
We don't need a map here, use simpler structures.
2020-02-27 12:00:03 +03:00
Roman Khimov
80de208a68 transaction: unbind GroupInputsByPrevHash from Transaction
Which allows to use it for ClaimTX. Make it also a bit more efficient as maps
are expensive.
2020-02-27 12:00:03 +03:00
Roman Khimov
b1b184f77f
Merge pull request #692 from nspcc-dev/persist-timer-fix
core: reschedule the next persist in a second from previous one
2020-02-26 10:34:40 +03:00
Roman Khimov
c258adb532 core: move spent coin management out of the inner storeBlock loop
prevHash == input.PrevHash, so make less DB accesses and more real work. Fix
some bugs along the way:
 * spentCoins structure may already be present in the DB when persisting TX,
   there is nothing wrong with that and we shouldn't overwrite it
 * it's only used for NEO and only to check for claim validity. Thus, when
   processing claim tx the corresponding spentCoins should always be present
   in the DB
2020-02-25 17:21:09 +03:00
Roman Khimov
36c6b6af14 core: put unspents get/put one loop above in storeBlock
Everywhere in this code prevHash == input.PrevHash, thus we can easily move
some common code out of the loop saving on DB accesses and
serialization/deserialization.
2020-02-24 18:53:10 +03:00
Roman Khimov
5c9681508b transaction: strip off a layer of redirection from ClaimTX.Claims
We don't need a pointer here and this change makes this field compatible with
Transaction.Inputs which is useful in many scenarios.
2020-02-24 18:22:27 +03:00
Roman Khimov
ed9e9e8590 storage: add locking into (*MemoryStore).Seek
It was missing there.
2020-02-24 17:51:50 +03:00
Roman Khimov
b49808b766 core: reschedule the next persist in a second from previous one
It makes little sense queueing up several persistence goroutines (or actually
even running them concurrently).
2020-02-24 17:17:25 +03:00
Roman Khimov
d5e6028c13 core: add validator public key into the list of hashes to verify
That should be done for Enrollment transactions.
2020-02-21 20:04:16 +03:00
Roman Khimov
a903147b60 keys: rename Signature to GetScriptHash, make it return Uint160
Signature itself wasn't used at all and its name is very misleading, Uint160
script hash is way more useful.
2020-02-19 12:19:03 +03:00
Roman Khimov
eb11e5fb11 core: implement basic policying support, fix #370
Implement mempool and consensus block creation policies, almost the same as
SimplePolicy plugin for C# node provides with two caveats:
 * HighPriorityTxType is not configured and hardcoded to ClaimType
 * BlockedAccounts are not supported

Other than that it allows us to run successfuly as testnet CN, previously our
proposals were rejected because we were proposing blocks with oversized
transactions (that are rejected by PoolTx() now).

Mainnet and testnet configuration files are updated accordingly, but privnet
is left as is with no limits.

Configuration is currently attached to the Blockchain and so is the code that
does policying, it may be moved somewhere in the future, but it works for
now.
2020-02-19 12:19:02 +03:00
Roman Khimov
22f5667530 mempool: return fee along with tx when requesting tx
Users of GetVerifiedTransactions() don't want to recalculate tx fee and it's
nice to have it returned from TryGetValue() also sometimes.
2020-02-19 12:19:02 +03:00
Roman Khimov
06daeb44f3 core: make IsLowPriority work with pre-calculated fee
Don't recalculate it again and again.
2020-02-19 12:19:02 +03:00
Roman Khimov
a4294f4b5f core: export UtilityTokenID and GoverningTokenID
Both are very useful outside of the core, this change also makes respective
transactions initialize with the package as they don't depend on any kind of
input and it makes no sense recreating them again and again on every use.
2020-02-19 12:13:27 +03:00
Roman Khimov
b41a5239c6 core: always sort the result of GetValidators
It wasn't sorted when all validators were elected. There is also no need to do
`Unique()` on the result because validators are distinguished by the key, so
no two registered validators can have the same key.
2020-02-18 20:36:22 +03:00
Evgenii Stratonikov
a2f012e589 core: deploy new smart-contract in test blocks
Also put smart-contract sources in testdata and
add invocation TX with APPCALL and storage.Put.
2020-02-18 11:59:28 +03:00
Anna Shaleva
2b9e63c511 core: add func to generate RPC test data
newRPCTestChain function generates test chain "testblocks.acc" for RPC
tests
2020-02-17 23:54:25 +03:00
Roman Khimov
dfb38e1da1
Merge pull request #649 from nspcc-dev/feature/dump_storage
cli: add possibility to dump storage changes on restore
2020-02-16 23:50:31 +03:00
Roman Khimov
b5453ad24f
Merge pull request #672 from nspcc-dev/fix-sigsegv-on-reference-check
core: deal with bad transaction Inputs
2020-02-14 13:23:42 +03:00
Roman Khimov
0fceaf12d2 core: deal with bad transaction Inputs
They may not really have a reference. Fixes #671.
2020-02-14 09:42:10 +03:00
Roman Khimov
992ccc0b26
Merge pull request #664 from nspcc-dev/sort-validator-keys
core: sort GetValidators() result
2020-02-13 21:46:26 +03:00
Evgeniy Kulikov
8d410a6659
Add JSON tags for RPC response types
- Attribute should have 2 fields (usage, data)
- VOut should have 4 (5) fields (asset, value, address, n)
- Script should have 2 fields (invocation, verification)
2020-02-13 19:06:54 +03:00
Roman Khimov
d4cd134aef core: sort GetValidators() result
As C# node does it. Technically it's only needed for consensus and could be
implemented in the appropriate package, but for better compatibility with C#
node we're better returning it sorted right here.
2020-02-13 17:07:27 +03:00
Roman Khimov
26af9200b0 core: refactor voting processing, fix pending tx subtractions
Deduplicate code and add missing token check for subtractions that are being
done in GetValidators() for pending transactions.
2020-02-13 13:07:34 +03:00
Roman Khimov
c8a248596e core: introduce ValidatorsCount, make a proper count
We were completely lacking ValidatorsCount that is supposed to track the
number of votes with particular count of consensus nodes which in theory can
change the number of active consensus nodes (if it ever to exceed the number
of standby validators), so we were not producing the right count and based on
that not giving the right set of validators.

Fixes #512.
2020-02-13 13:07:34 +03:00
Roman Khimov
357b675090 core: don't choose duplicating standby validators
We may already have chosen some of standby validators, so don't repeat.
2020-02-13 13:07:34 +03:00
Roman Khimov
f9c6a0d77c core: make voting choose the best validators
They need to be sorted in order for voting system to work.
2020-02-13 13:07:34 +03:00
Roman Khimov
2ff7ceb8cf core: properly update validator votes for Account descriptor
ValidatorsCount is not implemented yet, but we need to update validators
properly, adding account's vote to their votes count.
2020-02-13 13:07:34 +03:00
Roman Khimov
8310b76d28 core: properly save votes when processing Account descriptor
They were ignored basically.
2020-02-13 13:07:34 +03:00
Roman Khimov
807309a0de core/state: only drop unregistered non-voted validators
Simple as that: UnregisteredAndHasNoVotes != !RegisteredAndHasVotes

Registered validators should stay in the DB, we might be in the process of
updating votes for them and that starts with subtraction.
2020-02-13 13:07:34 +03:00
Roman Khimov
38d0efc96c core: fix Registered state descriptor processing
It has simple boolean value encoded, no `strconv` needed (it actually chokes
on such input).
2020-02-12 21:27:40 +03:00
Roman Khimov
803fb39bb0 transaction: fix wrong state descriptor serdes
Wrong field order.
2020-02-12 21:23:28 +03:00
Evgenii Stratonikov
0a894db7f8 storage: add Exists flag to KeyValue in batch
Set Exists flag if an item with the specified key was already
present in storage before persisting.
2020-02-12 12:16:31 +03:00
Evgenii Stratonikov
b1d9e1132d cli: dump storage changes into JSON files 2020-02-12 12:16:31 +03:00
Evgenii Stratonikov
fb9af98179 storage: implement GetBatch() to view storage changes
GetBatch returns changes to be persisted.
2020-02-12 12:16:31 +03:00
Roman Khimov
38de2af073
Merge pull request #655 from nspcc-dev/fix-mempool-deadlock-on-reverification
core: get transactions from dao when mempool should not be used
2020-02-11 13:34:23 +03:00
Roman Khimov
292051d8df core: get transactions from dao when mempool should not be used
All of these places deal strictly with the chain and shouldn't ever be
bothered with mempool. It also fixes a deadlock on reverification of
non-standard tx:

1 @ 0x42f62f 0x43fbe9 0x43fbbf 0x43f95d 0x967059 0x966f66 0x972c7c 0x974e13 0x97a5d9 0x97bdf0 0x976147 0x966cc0 0x970f70 0x96c8cb 0x9ba858 0x45ca51
	0x43f95c	sync.runtime_SemacquireMutex+0x3c							/usr/local/go/src/runtime/sema.go:71
	0x967058	sync.(*RWMutex).RLock+0x128								/usr/local/go/src/sync/rwmutex.go:50
	0x966f65	github.com/CityOfZion/neo-go/pkg/core/mempool.(*Pool).TryGetValue+0x35			/go/src/github.com/CityOfZion/neo-go/pkg/core/mempool/mem_pool.go:229
	0x972c7b	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).GetTransaction+0x4b			/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:782
	0x974e12	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).References+0x132			/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:944
	0x97a5d8	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).GetScriptHashesForVerifying+0x58	/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:1410
	0x97bdef	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).verifyTxWitnesses+0x4f		/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:1545
	0x976146	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).isTxStillRelevant+0x216		/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:1067
	0x966cbf	github.com/CityOfZion/neo-go/pkg/core/mempool.(*Pool).RemoveStale+0xff			/go/src/github.com/CityOfZion/neo-go/pkg/core/mempool/mem_pool.go:208
	0x970f6f	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).storeBlock+0x2ecf			/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:614
	0x96c8ca	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).AddBlock+0xea			/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:308
	0x9ba857	github.com/CityOfZion/neo-go/pkg/network.(*blockQueue).run+0x157			/go/src/github.com/CityOfZion/neo-go/pkg/network/blockqueue.go:48
2020-02-10 18:53:22 +03:00
Anna Shaleva
9b03f968b1 core: fixe FreeGasLimit constraint
Lack of FreeGasLimit in privnet leads to gas limit exceeding in case of transactions with small amount of GAS to be used for invoke operation (< real cost of the transaction). Solution: Fixed constraint in case when FreeGasLimit == 0. So now we are able to perform transactions in privnet with FreeGasLimit = 0 for free.
2020-02-10 18:07:19 +03:00
Roman Khimov
9b9adb28c1
Merge pull request #642 from nspcc-dev/feature/emit
vm, compiler: move Emit* functions to a separate package

Closes #449, #534.
2020-02-08 15:56:48 +03:00
Roman Khimov
7ccf7974b6 core: substitute bad notifications with error messages
Tesnet sync failed with:
Feb 07 00:04:19 nodoka neo-go[1747]: 2020-02-07T00:04:19.838+0300        WARN        blockQueue: failed adding block into the blockchain        {"error": "failed to store notifications: not supported", "blockHeight": 713984, "nextIndex": 713985}

because some (not so) smart contract emitted a notification with an
InteropItem inside.
2020-02-07 15:34:59 +03:00
Evgenii Stratonikov
8243a8b3a7 emit: use io.BinWriter instead of bytes.Buffer 2020-02-06 18:45:37 +03:00
Evgenii Stratonikov
4d8a3a359b vm: move Emit* functions to a separate package
Also strip 'Emit' prefix because 'emit' is now
in the package name.
2020-02-06 18:45:37 +03:00
Roman Khimov
ab14a4619d
Merge pull request #647 from nspcc-dev/fix-mempool-and-chain-locking
Fix mempool and chain locking

This allows us easily make 1000 Tx/s in 4-nodes privnet, fixes potential
double spends and improves mempool testing coverage.
2020-02-06 18:43:29 +03:00
Evgenii Stratonikov
9204b95d5b core: fix GAS price definitions 2020-02-06 18:19:08 +03:00
Roman Khimov
18695e660b mempool: drop RemoveOverCapacity(), handle it right in the Add()
Simplifies things a lot and removes useless code.
2020-02-06 17:50:11 +03:00
Roman Khimov
684cbf5bac mempool: make it almost 100% test-covered 2020-02-06 17:50:11 +03:00
Roman Khimov
6672f4b26f mempool: iterate over slice in GetVerifiedTransactions()
It's more efficient and keeps transactions sorted by priority.
2020-02-06 16:59:20 +03:00
Roman Khimov
e97396e56c mempool: test addition of conflicting tx to the pool 2020-02-06 15:57:20 +03:00
Roman Khimov
1133bbe584 mempool: remove unverified transactions pool
Our mempool only contains valid verified transactions all the time, it never
has any unverified ones. Unverified pool made some sense for quick unverifying
after the new block acceptance (and gradual background reverification), but
reverification needs some non-trivial locking between blockchain and mempool
and internal mempool state locking (reverifying tx and moving it between
unverified and verified pools must be atomic). But our current reverification
is fast enough (and has all the appropriate locks), so bothering with
unverified pool makes little sense.
2020-02-06 15:45:16 +03:00
Roman Khimov
b675903f52 mempool/core: redesign mempool dances on block acceptance
We not only need to remove transactions stored in the block, but also
invalidate some potential double spends caused by these transactions. Usually
new block contains a substantial number of transactions from the pool, so it's
easier to make one pass over it only keeping valid items rather than remove
them one by one and make an additional pass to recheck inputs/witnesses.
2020-02-06 15:45:16 +03:00
Roman Khimov
b567ce86ac mempool: implement insertion to sorted slice
Which is way faster than sort.Sort'ing things all the time.
2020-02-06 15:44:18 +03:00
Roman Khimov
35183b6dba mempool: reverse the order of sorted slice
Chopping off the last element of the slice if way easier than doing it with
the first one.
2020-02-06 15:44:16 +03:00
Roman Khimov
794027a90b mempool: use one slice for both priorities
It doesn't harm as we have transactions naturally ordered by fee anyway and it
makes managing them a little easier. This also makes slices store item itself
instead of pointers to it which reduces the pressure on the memory subsystem.
2020-02-06 15:43:45 +03:00
Roman Khimov
325bea3fa9 mempool: cache Feer invocation results in the item
They shouldn't depend on the chain state and for the same transaction they
should always produce the same result. Thus, it makes no sense recalculating
them over and over again.
2020-02-06 15:41:56 +03:00
Roman Khimov
e01bfeeb4d mempool: remove lock indirection from the Pool
After the f0bb886be3 with all methods of Pool
being pointer-based it makes no sense having this lock as a pointer.
2020-02-06 15:41:56 +03:00
Roman Khimov
a928ad9cfa mempool: make item an internal thing of mempool package
Nobody outside should care about these details, mempool operates on
transactions and that's it.
2020-02-06 15:41:56 +03:00
Roman Khimov
f0e3a31bc8 mempool: fix appending to sorted pools
Appending and not changing the real Items is utterly wrong.
2020-02-06 15:41:56 +03:00
Roman Khimov
f9963cca37 core: short-circuit verifyInputs when there are no inputs 2020-02-06 15:41:56 +03:00
Roman Khimov
70b3839fd0 core/mempool: fix AddBlock and tx pooling concurrency issues
Eliminate races between tx checks and adding them to the mempool, ensure the
chain doesn't change while we're working with the new tx. Ensure only one
block addition attempt could be in progress.
2020-02-06 15:41:52 +03:00
Roman Khimov
f0bb886be3 mempool: make all methods pointer methods
Makes no sense copying the Pool around.
2020-02-04 17:36:11 +03:00
Roman Khimov
388fed06e5 transaction: forbid serializing invalid transactions
Transaction that has no data is invalid and can't be serialized, so throw an
error if someone tries to.
2020-02-03 16:33:03 +03:00
Evgeniy Kulikov
eb59460032
Merge pull request #614 from nspcc-dev/configurable-mempool-size
config: add MemPoolSize configuration parameter
2020-01-23 13:25:27 +03:00
Roman Khimov
d6784e8f45 config: add MemPoolSize configuration parameter
Allow tuning/experimenting with mempool.
2020-01-23 13:10:44 +03:00
Evgenii Stratonikov
303b82cc44 core: set free GAS limit via config 2020-01-22 14:32:21 +03:00
Evgenii Stratonikov
4718375db1 rpc: return amount of consumed GAS in invoke* 2020-01-22 14:31:57 +03:00
Evgenii Stratonikov
4eae55143f core: restrict GAS available for invocation transactions
There are 10 GAS available for free plus any amount of GAS
attached to a transaction.
2020-01-22 14:31:54 +03:00
Roman Khimov
32213b1454
Merge pull request #601 from nspcc-dev/refactoring/core
core: refactor out Block, BlockBase and Header, closes #597.
2020-01-20 16:19:20 +03:00
Roman Khimov
46b82b4fb5 keys: don't return error from PrivateKey.Sign
As it can't ever happen.
2020-01-17 17:00:30 +03:00
Evgenii Stratonikov
28183b81d6 mempool: simplify names of exported types
With the move to a separate package, naming can be simplified:
MemPool -> Pool, PoolItem -> Item, PoolItems -> Items.
2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
fed6fba9b6 core: refactor out MemPool 2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
79bceb3e40 block: use require in tests 2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
489b88afbb block: rename BlockBase to Base 2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
63c56cca5c core: refactor out Block, BlockBase and Header structs
See #597.
2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
9dc5571327 core,rpc: close Blockchain in tests
If blockchain is not closed, logging in defer can occur
after test has finished, which will lead to a panic with
"Log in goroutine after Test* has completed".
2020-01-10 11:47:56 +03:00
Evgenii Stratonikov
637c99eda7 core: simplify newInteropContext 2020-01-10 11:14:29 +03:00
Evgenii Stratonikov
aecdf470e7 cli,pkg: use zap.Logger 2020-01-10 11:14:27 +03:00
Evgenii Stratonikov
9c79684516 storage: panic on error in boltdb.Seek
Error in Seek means something is terribly wrong (e.g. db was not opened) and
error drop is not the right thing to do, because caller
will continue working with the wrong view.
2020-01-09 15:17:35 +03:00
Roman Khimov
c306d2f07c
Merge pull request #581 from nspcc-dev/feat/storage_find
core: implement Neo.Storage.Find interop, closes #422.
2019-12-26 15:46:46 +03:00
Evgenii Stratonikov
48cf4f4a84 core: implement Neo.Storage.Find interop 2019-12-26 15:21:41 +03:00
Roman Khimov
f8887b63ba
Merge pull request #578 from nspcc-dev/fix/sign
consensus: verify payloads correctly, closes #555.
2019-12-26 13:07:40 +03:00
Evgenii Stratonikov
714c466c2c core: add ScriptFromWitness function
Extracting verification script from witness became a common task.
This commit adds such a possibility.
2019-12-26 10:49:56 +03:00
Roman Khimov
9145855d2c
Merge pull request #579 from nspcc-dev/refactor-crypto
This moves some functionality into micro-packages, improves testing,
unexports some code and fixes bugs along the way.
2019-12-25 18:12:50 +03:00
Roman Khimov
b246653f62 address: rename functions as per #579 comments
Make them more clear to understand.
2019-12-25 17:34:18 +03:00
Roman Khimov
e685e9bf9a address: move into its own package
Doesn't really belong to the crypto.
2019-12-25 15:22:02 +03:00
Roman Khimov
ee28fb08f6 crypto: move merkle tree into the hash package
It's all about hashes, so it makes sense putting it there.
2019-12-25 11:28:59 +03:00
Vsevolod Brekelov
7f1844a39e transaction: add unmarshaling for TXType 2019-12-25 01:44:37 +03:00
Evgenii Stratonikov
5bc32b523a vm: implement Neo.Iterator.* interops 2019-12-24 10:21:47 +03:00
Evgenii Stratonikov
3ff7fd5262 vm: implement Neo.Enumerator.* interops 2019-12-24 10:21:46 +03:00
Roman Khimov
bf84e1f2fb core: cache top block
Turns out, our dApps use it a lot and we were going to the DB to get it which
is a useless waste of time. Technically we could also remove blockHeight here,
but not doing it at the moment as it's more involved.

It eliminates this time waste from the pprof graph, but doesn't change 1.4M ->
1.5M 100K mainnet block import test case in any noticeable way.
2019-12-23 19:18:12 +03:00
Roman Khimov
deb76c5199 core: add negative non-interop test for all interops
Try to feed some garbage into these functions.
2019-12-23 16:14:43 +03:00
Roman Khimov
76d69ee5ee core: restore proper interops ordering 2019-12-23 16:14:43 +03:00
Roman Khimov
c287a9e93c core: implement Neo.Witness.GetVerificationScript interop 2019-12-23 16:14:43 +03:00
Roman Khimov
9d638d0fee core: implement Neo.InvocationTransaction.GetScript 2019-12-23 16:14:43 +03:00
Roman Khimov
a7457d08a1 vm/core: add ID support for SYSCALL, redo interop registration
This solves two problems:
 * adds support for shortened SYSCALL form that uses IDs (similar to #434, but
   for NEO 2.0, supporting both forms), which is important for compatibility
   with C# node and mainnet chain that uses it from some height
 * reworks interop plugging to use callbacks rather than appending to the map,
   these map mangling functions are clearly visible in the VM profiling
   statistics and we want spawning a VM to be fast, so it makes sense
   optimizing it. This change moves most of the work to the init() phase
   making VM setup cheaper.

Caveats:
 * InteropNameToID accepts `[]byte` because that's the thing we have in
   SYSCALL processing and that's the most often usecase for it, it leads to
   some conversions in other places but that's acceptable because those are
   either tests or init()
 * three getInterop functions are: `getDefaultVMInterop`, `getSystemInterop`
   and `getNeoInterop`

Our 100K (1.4M->1.5M) block import time improves by ~4% with this change.
2019-12-19 13:35:42 +03:00
Evgenii Stratonikov
884779e501 *: goimports 2019-12-17 14:51:28 +03:00
Roman Khimov
7a5e995030 core: add MemPool Verify() test 2019-12-14 00:05:45 +03:00
Roman Khimov
5d68f88196 core: fix and speed up mempool Verify()
First of all, it was wrong, it was not checking for inputs really, it compared
tx hashes for some reason, second, when it did compare inputs it compared only
the PrevIndex part of them which is also wrong.

Also, there is absolutely no reason to go through GetVerifiedTransactions()
here, we don't need this copy of pointers and it can also be outdated by the
time we're to finish our check.

Before:
BenchmarkTXPerformanceTest-4
    5000            485506 ns/op           65886 B/op        409 allocs/op
ok      github.com/CityOfZion/neo-go/integration        3.212s

After:
enchmarkTXPerformanceTest-4
    5000            371104 ns/op           44367 B/op        408 allocs/op
ok      github.com/CityOfZion/neo-go/integration        2.712s
2019-12-13 23:23:33 +03:00
Roman Khimov
e631e75718 core: optimize GetVerifiedTransactions()
This simple change improves our BenchmarkTXPerformanceTest by 14%, just
because we don't waste time on reallocations during append().

Before:
   10000            439754 ns/op          218859 B/op        428 allocs/op
ok      github.com/CityOfZion/neo-go/integration        5.423s

After:
   10000            369833 ns/op           87209 B/op        412 allocs/op
ok      github.com/CityOfZion/neo-go/integration        4.612s
2019-12-13 19:57:17 +03:00
Roman Khimov
a9401e2ec7 core: do not write new dao elements into DB
It's useless work being done before it's actually needed. These (updated with
new values) are going to be written with some kind of Put anyway, so writing
them here is just a waste of time.
2019-12-13 19:57:17 +03:00
Roman Khimov
6896b40dee state: use open-coded array (de)serialization for accounts
We're spending a lot of time here, 100K blocks import starting at 1.4M, before
this patch:
real    4m17,748s
user    6m23,316s
sys     0m37,866s

After:
real    3m54,968s
user    5m56,547s
sys     0m39,398s

9% is quite a substantial improvement to justify this change.
2019-12-13 19:57:17 +03:00
Roman Khimov
0d0a27d271 core: add cachedDao to cache accounts and contracts
Importing 100K blocks starting at 1.4M, before this patch:
real    6m0,356s
user    8m52,293s
sys     0m47,372s

After this patch:
real    4m17,748s
user    6m23,316s
sys     0m37,866s

Almost 30% better.
2019-12-13 19:57:17 +03:00
Roman Khimov
212cf44e26 core: get data from dao in interops
It's more logically correct as Blockchain can be several transactions behind
in terms of its state.
2019-12-13 17:17:14 +03:00
Roman Khimov
0abd55c2c2 vm: add cached ScriptHash() to Context
Avoid recalculating it over and over again in interop.
2019-12-13 17:05:03 +03:00
Roman Khimov
09223236f9 core: add Persist() to dao()
Hide its internals a little, which is gonna be useful for future composition.
2019-12-12 21:17:13 +03:00
Roman Khimov
eb1749d190 core: remove persisting from GetValidators()
It's a getter function and even though it's quite fancy with its transactions
processing (for consensus operation) it shouldn't ever change the state of the
Blockchain. If we're to change anything here these changes may conflict with
the actual block processing later or may lead to broken state (if transactions
won't be approved for some reason).
2019-12-12 21:14:33 +03:00
Roman Khimov
31dbc35279 core: gofmt dao_test.go 2019-12-12 21:05:36 +03:00
Roman Khimov
c5ed3b788b core: simplify dao creation with newDao() 2019-12-12 21:04:55 +03:00
Roman Khimov
8b3080b972 io: rename Read/WriteBytes to Read/WriteB
go vet is not happy about them:
  pkg/io/binaryReader.go:92:21: method ReadByte() byte should have signature ReadByte() (byte, error)
  pkg/io/binaryWriter.go:75:21: method WriteByte(u8 byte) should have signature WriteByte(byte) error
2019-12-12 20:19:50 +03:00
Roman Khimov
54d888ba70 io: add type-specific read/write methods
This seriously improves the serialization/deserialization performance for
several reasons:
 * no time spent in `binary` reflection
 * no memory allocations being made on every read/write
 * uses fast ReadBytes everywhere it's appropriate

It also makes Fixed8 Serializable just for convenience.
2019-12-12 20:19:50 +03:00
Roman Khimov
89d7f6d26e core/tx: microoptimize block/tx hashing
Don't hash the data twice.
2019-12-12 18:01:30 +03:00
Vsevolod Brekelov
c93a8d2bc4 core: extracted same logic to separate methods 2019-12-11 13:14:45 +03:00
Vsevolod Brekelov
c1f39d5c7b internal: moved testutil method to internal package 2019-12-11 13:14:43 +03:00
Vsevolod Brekelov
2d42b14a1d core: renames entities-> state and removed State prefix 2019-12-11 13:14:18 +03:00
Vsevolod Brekelov
8809fe437d core: unit tests for dao 2019-12-11 13:13:51 +03:00
Vsevolod Brekelov
c0e59ebd4e core: unit tests for entities 2019-12-11 13:05:31 +03:00
Vsevolod Brekelov
c7ac4b6dff core: fix encoding and decoding for notification event 2019-12-11 13:05:31 +03:00
Vsevolod Brekelov
ec17654986 core: refactoring blockchain state and storage
add dao which takes care about all CRUD operations on storage
remove blockchain state since everything is stored on change
remove storage operations from structs(entities)
move structs to entities package
2019-12-11 13:05:31 +03:00
Roman Khimov
d0f9a28196 vm/core: improve block import speed with PublicKey caching
This change (closely related to the neo-project/neo#1321 proposal) speeds up
1.4M mainnet blocks import by 30%. Basically, we're eliminating key decoding
for block's multisignature that has the same keys most of the time.

Things I don't like about this patch:
 * yet another parameter for verifyHashAgainstScript()
 * vm keys are not copied in/out

But it's rather simple and solves the problem for this particular case, so I
think it's worth it.
2019-12-10 19:13:29 +03:00
Roman Khimov
f1856bfa8b core/tx: remove publickey indirection from assets and txes
It makes very little sense having pointers here, these structures MUST have
some kind of key and this key is not gonna be wandering somewhere on its
own. Fixes a part of #519.
2019-12-09 18:33:04 +03:00
Roman Khimov
5b6c5af704 *: implement EncodeBinary with pointer receivers where appropriate
Everywhere except ParamType (which is just a byte), reduce copying things
around for no real reason.
2019-12-09 18:25:15 +03:00
Roman Khimov
7e371588a7 core/tx: remove one layer of indirection for scripts and inouts
It reduces heap pressure a little for these elements as we don't have to
allocate/free them individually. And they're directly tied to transactions or
block, not being shared or anything like that, so it makes little sense for
them to be pointer-based. It only makes building transactions a little easier,
but that's obviously a minor usecase.
2019-12-09 17:14:10 +03:00
Evgenii Stratonikov
fccb008594 io: implement ReadBytes() 2019-12-09 15:00:15 +03:00
Evgenii Stratonikov
838050f8b5 io: rename ReadBytes() to ReadVarBytes() 2019-12-09 15:00:15 +03:00
Roman Khimov
e4d821f32d
Merge pull request #546 from nspcc-dev/write-optimizations
Write optimizations
2019-12-06 19:40:38 +03:00
Roman Khimov
9992a98007 core: optimize balance utxo removal
One of my samples had 8 (out of 30) seconds spent here, but values of 100ms
are more typical. After this change it becomes invisible.
2019-12-06 18:46:24 +03:00
Roman Khimov
844491d365 *: use more efficient WriteBytes where appropriate
Before this patch on block import we could easily be spending more than 6
seconds out of 30 in Uint256 encoding for UnspentBalance, now it's completely
off the radar.
2019-12-06 18:22:21 +03:00
Evgenii Stratonikov
7179e4ba9f util: add LE suffix to Uint256 methods 2019-12-06 12:16:55 +03:00
Evgenii Stratonikov
57efad912c util: add LE suffix to Uint160 methods 2019-12-06 12:16:55 +03:00
Roman Khimov
8beb135829
Merge pull request #542 from nspcc-dev/fix-storage-interop-reads
core: fix wrong data being read in interops (part of #501)
2019-12-05 09:08:38 +03:00
Roman Khimov
c034aae378 core: fix wrong data being read in interops (part of #501)
When 74590551 introduced this code we had no proper caching layer, so there
were these strange fallbacks in the code. fc0031e5 should'd removed them, but
failed to do so, so do it now and fix processing of transactions that touch
storage for the same key (address) in the same block.
2019-12-04 19:51:57 +03:00
Vsevolod Brekelov
ff15a0acfd core: add trigger types as constants. Closes #509 2019-12-04 12:27:04 +03:00
Roman Khimov
31add423a8 core: replace open-coded PUSHBYTES64 with the one from opcode 2019-12-03 18:23:46 +03:00
Roman Khimov
138e125646 *: remove duplicate functions producing verification script
Drop wif.GetVerificationScript(), drop
smartcontract.CreateSignatureRedeemScript(), add GetVerificationScript()
directly to the PublicKey and use it everywhere.
2019-12-03 18:23:46 +03:00
Roman Khimov
8d4dd2d2e1 vm: move opcodes into their own package
This allows easier reuse of opcodes and in some cases allows to eliminate
dependencies on the whole vm package, like in compiler that only needs opcodes
and doesn't care about VM for any other purpose.

And yes, they're opcodes because an instruction is a whole thing with
operands, that's what context.Next() returns.
2019-12-03 18:22:14 +03:00
Roman Khimov
f48228ef7d
Merge pull request #467 from nspcc-dev/errcheck_297
This patchset closes #297 and #457.
2019-12-03 15:06:11 +03:00
Vsevolod Brekelov
03ff2976ed io: refactoring for using WriteVarBytes instead of WriteLE
goal is to be consistent with C# implementation.
For writing []byte WriteBytes used and for byte - WriteVarByte.
2019-12-03 13:49:33 +03:00
Roman Khimov
1c89c192ac core: optimize some accesses to unsortedTxn in mempool
These don't need a full lock, they only read things from maps.
2019-12-02 22:39:43 +03:00
Roman Khimov
0cd3493fa5 core: fix potential locking problem in mempool
I think it should fix this issue mentioned in the #526:

INFO[1456] blockchain persist completed                  blockHeight=63480 headerHeight=1810000 persistedBlocks=1 persistedKeys=4 took=740.7113ms
fatal error: concurrent map read and map write

goroutine 322 [running]:
runtime.throw(0xc8a6dc, 0x21)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc003473958 sp=0xc003473928 pc=0x42e282
runtime.mapaccess2(0xb706a0, 0xc0001893b0, 0xc0034739c8, 0xc0028704e0, 0x3)
        /usr/local/go/src/runtime/map.go:470 +0x278 fp=0xc0034739a0 sp=0xc003473958 pc=0x40dc08
github.com/CityOfZion/neo-go/pkg/core.MemPool.ContainsKey(0xc0001d0d20, 0xc0001893b0, 0xc0001893e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /neo-go/pkg/core/mem_pool.go:92 +0xcb fp=0xc003473a30 sp=0xc0034739a0 pc=0x9326db
github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).HasTransaction(0xc0001ca8c0, 0x49f0b45d430e441b, 0x553db79b7072821c, 0x28969518de11976, 0xba5100efddbe79d4, 0xc003473cd0)
        /neo-go/pkg/core/blockchain.go:803 +0xa1 fp=0xc003473b68 sp=0xc003473a30 pc=0x914b11
github.com/CityOfZion/neo-go/pkg/core.Blockchainer.HasTransaction-fm(0x49f0b45d430e441b, 0x553db79b7072821c, 0x28969518de11976, 0xba5100efddbe79d4, 0xc005a5d388)
        /neo-go/pkg/core/blockchainer.go:28 +0x46 fp=0xc003473ba8 sp=0xc003473b68 pc=0x997326
github.com/CityOfZion/neo-go/pkg/network.(*Server).handleInvCmd(0xc00018f680, 0xd9d980, 0xc00025e190, 0xc005a5d380, 0x0, 0x0)
        /neo-go/pkg/network/server.go:401 +0x3bb fp=0xc003473d38 sp=0xc003473ba8 pc=0x9924cb
github.com/CityOfZion/neo-go/pkg/network.(*Server).handleMessage(0xc00018f680, 0xd9d980, 0xc00025e190, 0xc007a0d050, 0x0, 0x0)
        /neo-go/pkg/network/server.go:582 +0x1ae fp=0xc003473da0 sp=0xc003473d38 pc=0x993bbe
github.com/CityOfZion/neo-go/pkg/network.(*TCPTransport).handleConn(0xc000228420, 0xd9b880, 0xc0001b6f00)
        /neo-go/pkg/network/tcp_transport.go:93 +0x202 fp=0xc003473fc8 sp=0xc003473da0 pc=0x996672
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc003473fd0 sp=0xc003473fc8 pc=0x45b3e1
created by github.com/CityOfZion/neo-go/pkg/network.(*TCPTransport).Dial
        /neo-go/pkg/network/tcp_transport.go:36 +0xb4

The problem is that we're modifying `unsortedTxn` under a reader lock.
2019-12-02 22:36:59 +03:00
Roman Khimov
bc5beb438f *: gofmt
Somehow these crept into the repository.
2019-12-02 10:10:50 +03:00
Roman Khimov
2c0ed62680 core: fix blockchain init with (2000*N + 1) header hashes in DB
Stored header count includes block number zero, so it's (height + 1).
2019-11-29 16:26:44 +03:00
Roman Khimov
2d41450ac9
Merge pull request #507 from nspcc-dev/dbft
network: plug in dBFT library
2019-11-27 15:22:40 +03:00
Roman Khimov
c5e2b3ce73
Merge pull request #523 from nspcc-dev/fix-interim-block-changes-store
Fix interim block changes store
2019-11-27 14:51:29 +03:00
Vsevolod
1a9e98caf3
Merge pull request #522 from nspcc-dev/fix-invalid-tx-type-handling
Fix invalid tx type handling
2019-11-27 13:40:52 +03:00
Roman Khimov
467c9c146d core: restore intermediate block-level store
Commit c80ee952a1 removed temporary store used
to contain changes of the block being processed. It's wrong in that the block
changes should be applied to the database in a single transaction so that
there wouldn't be any intermediate state observed from the outside (which is
possible now). Also, this made changes commiting persist them to the
underlying store effectively making our persist loop a no-op (and not
producing `persist completed` log lines that we love so much).
2019-11-27 13:28:13 +03:00
Roman Khimov
bb765ccfa7 core: gofmt 2019-11-27 13:27:47 +03:00
Roman Khimov
65bf0d01d5 transaction: don't hide decoding error by logging
Unknown transaction types are wrong and should be treated as errors.
2019-11-27 13:12:04 +03:00
Evgenii Stratonikov
fdd5276d3e network: plug in dBFT library 2019-11-27 10:57:22 +03:00
Evgenii Stratonikov
af61571ba0 core: do not return error in block.getHashableData 2019-11-26 15:27:37 +03:00
Roman Khimov
eb84ae49da
Merge pull request #508 from nspcc-dev/feature/getvalidators_420
core: Implement getvalidators interop
2019-11-22 18:15:08 +03:00
Vsevolod Brekelov
c03de88b41 core: unit tests blockchain state 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
d28f2dce05 core: unit test validator state 2019-11-21 15:41:49 +03:00