Commit graph

2182 commits

Author SHA1 Message Date
Evgenii Stratonikov
6add4f3e50 transaction: disallow negative outputs
Otherwise it is possible to make outputs which will sum
to the expected value, but steal GAS from some other account.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
9e9d59b49a core: set NetworkFee to 0 for Claim and Miner transactions
Claim tx have no GAS inputs and a positive output which
can lead to negative network fee.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
de5215a564 core: use GetAccountStateOrNew() in (*Blockchain).GetValidators()
Target of the transaction output may not yet exist in database.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
1b9968df67 core: optimize CalculateClaimable()
Because accumulated system fee is stored for every block,
it is easy to calculate sum with just to reads.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
258d8be1dd core: store all accumulated SystemFee with every block 2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
f69c8a763b core: store system fee together with block
Recalculating system fee can be rather costly if done
frequently.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
972e0d8ad1 core: add one more Contract tx to the test chain
When testing CLI it is useful to have some spent coins
on an account with a known key.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
45b8669b42 core: verify Claim transactions 2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
44792ed5f7 core: make test chain format compatible with mainnet
Write uint32 length before every block.
2020-03-02 18:01:49 +03:00
Evgenii Stratonikov
0e2a1f40ba core: add ContractTX to a testchain generator
Simple transfer from multisig account to the account
of one of the validators.
2020-03-02 18:00:00 +03:00
Evgenii Stratonikov
72d72296c3 core: implement (*Transaction).GetSignedPart()
Marshalling it and taking all but last byte violates incapsulation
and is just wrong in case transaction already contains any witnesses.
2020-03-02 18:00:00 +03:00
Evgenii Stratonikov
95d9f36c98 core: implement UnclaimedBalance tracking
To make it easy to get unclaimed coins for the
specified account they must be tracked together.
2020-03-02 18:00:00 +03:00
Evgenii Stratonikov
7095ec6c51 core: implement (*Blockchain).CalculateClaimable
Calculating amount of GAS that can be claimed is required
for getclaimable RPC.
2020-03-02 18:00:00 +03:00
Roman Khimov
252a9f2f31
Merge pull request #690 from nspcc-dev/feature/getapplicationlog
rpc: implement getapplicationlog RPC
2020-03-02 17:41:32 +03:00
Anna Shaleva
ff4384d7ff rpc: implement getapplicationlog RPC
Closes #500
2020-03-02 17:25:27 +03:00
Anna Shaleva
7d46404e2d smartcontract: turn trigger types into Type
1) Turn trigger types from byte constants into Type
2) Add auto-generated stringer for future purposes
2020-03-02 17:25:27 +03:00
Evgenii Stratonikov
a3dacd3b74 tests: replace t.Fatal with require where possible
This makes tests less verbose and unifies the style
they are written in.
2020-03-02 17:22:27 +03:00
Evgenii Stratonikov
66f96e3f32 core: shutdown Blockchain gracefully in tests 2020-03-02 17:22:27 +03:00
Evgenii Stratonikov
3a5224344e core: verify headers in AddHeaders()
Headers can be malformed so public methods should verify them
before adding.
2020-03-02 17:22:26 +03:00
Evgenii Stratonikov
357bb4ce41 core: get rid of global variables in tests
It can lead to unnecessary race conditions and is just
a bad practice.
2020-03-02 17:04:08 +03:00
Evgenii Stratonikov
8ca94e23c8 core: replace makeBlocks() with addBlocks() in tests
This simplifies tests a bit.
2020-03-02 17:04:08 +03:00
Evgenii Stratonikov
cbf26bac83 mempool: do not allocate new slice for verified transactions
Because transactions a iterated in an increasing order,
we can filter slice in-place.
2020-03-02 10:38:27 +03:00
Roman Khimov
3e2b490025 mempool: rework inputs verification, check Claim txes
Use more efficient check for Input and also check Claims to avoid double
claiming.
2020-02-27 13:43:35 +03:00
Roman Khimov
f329de73e8 mempool: fix Remove bug (dropping wrong tx!)
Uncovered by TestMemPoolAddRemoveWithInputsAndClaims (to be added later)
2020-02-27 13:43:35 +03:00
Roman Khimov
7d59fa0066
Merge pull request #696 from nspcc-dev/tx-verification-fixes
Claim and enrollment TX verification fixes
2020-02-27 12:45:36 +03:00
Roman Khimov
00d199ed85 core: fix NetworkFee calculation
Add doesn't change the variable state. Thanks to GolangCI for catching this.
2020-02-27 12:32:56 +03:00
Roman Khimov
7bea6e043e core: add double claim verification check 2020-02-27 12:32:56 +03:00
Roman Khimov
29586f2aa7 core: add duplicate Claim verification check
Refactor HaveInputsDuplicate() out of the core and Blockchain, it doesn't
depend on the Blockchain state. Make it more efficient.
2020-02-27 12:32:56 +03:00
Roman Khimov
9b1ad40273 core: drop GetScriptHashesForVerifyingClaim, fix Claim's ScriptHash getter
Claim transactions should _add_ claims scripthashes to the standard list, not
replace them. And this code is actually very reusable.
2020-02-27 12:32:56 +03:00
Roman Khimov
9f7018503a core/transaction: use new transaction.InOut for References
We don't need a map here, use simpler structures.
2020-02-27 12:00:03 +03:00
Roman Khimov
80de208a68 transaction: unbind GroupInputsByPrevHash from Transaction
Which allows to use it for ClaimTX. Make it also a bit more efficient as maps
are expensive.
2020-02-27 12:00:03 +03:00
Roman Khimov
b1b184f77f
Merge pull request #692 from nspcc-dev/persist-timer-fix
core: reschedule the next persist in a second from previous one
2020-02-26 10:34:40 +03:00
Roman Khimov
c258adb532 core: move spent coin management out of the inner storeBlock loop
prevHash == input.PrevHash, so make less DB accesses and more real work. Fix
some bugs along the way:
 * spentCoins structure may already be present in the DB when persisting TX,
   there is nothing wrong with that and we shouldn't overwrite it
 * it's only used for NEO and only to check for claim validity. Thus, when
   processing claim tx the corresponding spentCoins should always be present
   in the DB
2020-02-25 17:21:09 +03:00
Roman Khimov
36c6b6af14 core: put unspents get/put one loop above in storeBlock
Everywhere in this code prevHash == input.PrevHash, thus we can easily move
some common code out of the loop saving on DB accesses and
serialization/deserialization.
2020-02-24 18:53:10 +03:00
Roman Khimov
5c9681508b transaction: strip off a layer of redirection from ClaimTX.Claims
We don't need a pointer here and this change makes this field compatible with
Transaction.Inputs which is useful in many scenarios.
2020-02-24 18:22:27 +03:00
Roman Khimov
ed9e9e8590 storage: add locking into (*MemoryStore).Seek
It was missing there.
2020-02-24 17:51:50 +03:00
Roman Khimov
b49808b766 core: reschedule the next persist in a second from previous one
It makes little sense queueing up several persistence goroutines (or actually
even running them concurrently).
2020-02-24 17:17:25 +03:00
Roman Khimov
d5e6028c13 core: add validator public key into the list of hashes to verify
That should be done for Enrollment transactions.
2020-02-21 20:04:16 +03:00
Roman Khimov
a903147b60 keys: rename Signature to GetScriptHash, make it return Uint160
Signature itself wasn't used at all and its name is very misleading, Uint160
script hash is way more useful.
2020-02-19 12:19:03 +03:00
Roman Khimov
eb11e5fb11 core: implement basic policying support, fix #370
Implement mempool and consensus block creation policies, almost the same as
SimplePolicy plugin for C# node provides with two caveats:
 * HighPriorityTxType is not configured and hardcoded to ClaimType
 * BlockedAccounts are not supported

Other than that it allows us to run successfuly as testnet CN, previously our
proposals were rejected because we were proposing blocks with oversized
transactions (that are rejected by PoolTx() now).

Mainnet and testnet configuration files are updated accordingly, but privnet
is left as is with no limits.

Configuration is currently attached to the Blockchain and so is the code that
does policying, it may be moved somewhere in the future, but it works for
now.
2020-02-19 12:19:02 +03:00
Roman Khimov
22f5667530 mempool: return fee along with tx when requesting tx
Users of GetVerifiedTransactions() don't want to recalculate tx fee and it's
nice to have it returned from TryGetValue() also sometimes.
2020-02-19 12:19:02 +03:00
Roman Khimov
06daeb44f3 core: make IsLowPriority work with pre-calculated fee
Don't recalculate it again and again.
2020-02-19 12:19:02 +03:00
Roman Khimov
a4294f4b5f core: export UtilityTokenID and GoverningTokenID
Both are very useful outside of the core, this change also makes respective
transactions initialize with the package as they don't depend on any kind of
input and it makes no sense recreating them again and again on every use.
2020-02-19 12:13:27 +03:00
Roman Khimov
b41a5239c6 core: always sort the result of GetValidators
It wasn't sorted when all validators were elected. There is also no need to do
`Unique()` on the result because validators are distinguished by the key, so
no two registered validators can have the same key.
2020-02-18 20:36:22 +03:00
Evgenii Stratonikov
a2f012e589 core: deploy new smart-contract in test blocks
Also put smart-contract sources in testdata and
add invocation TX with APPCALL and storage.Put.
2020-02-18 11:59:28 +03:00
Anna Shaleva
2b9e63c511 core: add func to generate RPC test data
newRPCTestChain function generates test chain "testblocks.acc" for RPC
tests
2020-02-17 23:54:25 +03:00
Roman Khimov
dfb38e1da1
Merge pull request #649 from nspcc-dev/feature/dump_storage
cli: add possibility to dump storage changes on restore
2020-02-16 23:50:31 +03:00
Roman Khimov
b5453ad24f
Merge pull request #672 from nspcc-dev/fix-sigsegv-on-reference-check
core: deal with bad transaction Inputs
2020-02-14 13:23:42 +03:00
Roman Khimov
0fceaf12d2 core: deal with bad transaction Inputs
They may not really have a reference. Fixes #671.
2020-02-14 09:42:10 +03:00
Roman Khimov
992ccc0b26
Merge pull request #664 from nspcc-dev/sort-validator-keys
core: sort GetValidators() result
2020-02-13 21:46:26 +03:00
Evgeniy Kulikov
8d410a6659
Add JSON tags for RPC response types
- Attribute should have 2 fields (usage, data)
- VOut should have 4 (5) fields (asset, value, address, n)
- Script should have 2 fields (invocation, verification)
2020-02-13 19:06:54 +03:00
Roman Khimov
d4cd134aef core: sort GetValidators() result
As C# node does it. Technically it's only needed for consensus and could be
implemented in the appropriate package, but for better compatibility with C#
node we're better returning it sorted right here.
2020-02-13 17:07:27 +03:00
Roman Khimov
26af9200b0 core: refactor voting processing, fix pending tx subtractions
Deduplicate code and add missing token check for subtractions that are being
done in GetValidators() for pending transactions.
2020-02-13 13:07:34 +03:00
Roman Khimov
c8a248596e core: introduce ValidatorsCount, make a proper count
We were completely lacking ValidatorsCount that is supposed to track the
number of votes with particular count of consensus nodes which in theory can
change the number of active consensus nodes (if it ever to exceed the number
of standby validators), so we were not producing the right count and based on
that not giving the right set of validators.

Fixes #512.
2020-02-13 13:07:34 +03:00
Roman Khimov
357b675090 core: don't choose duplicating standby validators
We may already have chosen some of standby validators, so don't repeat.
2020-02-13 13:07:34 +03:00
Roman Khimov
f9c6a0d77c core: make voting choose the best validators
They need to be sorted in order for voting system to work.
2020-02-13 13:07:34 +03:00
Roman Khimov
2ff7ceb8cf core: properly update validator votes for Account descriptor
ValidatorsCount is not implemented yet, but we need to update validators
properly, adding account's vote to their votes count.
2020-02-13 13:07:34 +03:00
Roman Khimov
8310b76d28 core: properly save votes when processing Account descriptor
They were ignored basically.
2020-02-13 13:07:34 +03:00
Roman Khimov
807309a0de core/state: only drop unregistered non-voted validators
Simple as that: UnregisteredAndHasNoVotes != !RegisteredAndHasVotes

Registered validators should stay in the DB, we might be in the process of
updating votes for them and that starts with subtraction.
2020-02-13 13:07:34 +03:00
Roman Khimov
38d0efc96c core: fix Registered state descriptor processing
It has simple boolean value encoded, no `strconv` needed (it actually chokes
on such input).
2020-02-12 21:27:40 +03:00
Roman Khimov
803fb39bb0 transaction: fix wrong state descriptor serdes
Wrong field order.
2020-02-12 21:23:28 +03:00
Evgenii Stratonikov
0a894db7f8 storage: add Exists flag to KeyValue in batch
Set Exists flag if an item with the specified key was already
present in storage before persisting.
2020-02-12 12:16:31 +03:00
Evgenii Stratonikov
b1d9e1132d cli: dump storage changes into JSON files 2020-02-12 12:16:31 +03:00
Evgenii Stratonikov
fb9af98179 storage: implement GetBatch() to view storage changes
GetBatch returns changes to be persisted.
2020-02-12 12:16:31 +03:00
Roman Khimov
38de2af073
Merge pull request #655 from nspcc-dev/fix-mempool-deadlock-on-reverification
core: get transactions from dao when mempool should not be used
2020-02-11 13:34:23 +03:00
Roman Khimov
292051d8df core: get transactions from dao when mempool should not be used
All of these places deal strictly with the chain and shouldn't ever be
bothered with mempool. It also fixes a deadlock on reverification of
non-standard tx:

1 @ 0x42f62f 0x43fbe9 0x43fbbf 0x43f95d 0x967059 0x966f66 0x972c7c 0x974e13 0x97a5d9 0x97bdf0 0x976147 0x966cc0 0x970f70 0x96c8cb 0x9ba858 0x45ca51
	0x43f95c	sync.runtime_SemacquireMutex+0x3c							/usr/local/go/src/runtime/sema.go:71
	0x967058	sync.(*RWMutex).RLock+0x128								/usr/local/go/src/sync/rwmutex.go:50
	0x966f65	github.com/CityOfZion/neo-go/pkg/core/mempool.(*Pool).TryGetValue+0x35			/go/src/github.com/CityOfZion/neo-go/pkg/core/mempool/mem_pool.go:229
	0x972c7b	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).GetTransaction+0x4b			/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:782
	0x974e12	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).References+0x132			/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:944
	0x97a5d8	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).GetScriptHashesForVerifying+0x58	/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:1410
	0x97bdef	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).verifyTxWitnesses+0x4f		/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:1545
	0x976146	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).isTxStillRelevant+0x216		/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:1067
	0x966cbf	github.com/CityOfZion/neo-go/pkg/core/mempool.(*Pool).RemoveStale+0xff			/go/src/github.com/CityOfZion/neo-go/pkg/core/mempool/mem_pool.go:208
	0x970f6f	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).storeBlock+0x2ecf			/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:614
	0x96c8ca	github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).AddBlock+0xea			/go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:308
	0x9ba857	github.com/CityOfZion/neo-go/pkg/network.(*blockQueue).run+0x157			/go/src/github.com/CityOfZion/neo-go/pkg/network/blockqueue.go:48
2020-02-10 18:53:22 +03:00
Anna Shaleva
9b03f968b1 core: fixe FreeGasLimit constraint
Lack of FreeGasLimit in privnet leads to gas limit exceeding in case of transactions with small amount of GAS to be used for invoke operation (< real cost of the transaction). Solution: Fixed constraint in case when FreeGasLimit == 0. So now we are able to perform transactions in privnet with FreeGasLimit = 0 for free.
2020-02-10 18:07:19 +03:00
Roman Khimov
9b9adb28c1
Merge pull request #642 from nspcc-dev/feature/emit
vm, compiler: move Emit* functions to a separate package

Closes #449, #534.
2020-02-08 15:56:48 +03:00
Roman Khimov
7ccf7974b6 core: substitute bad notifications with error messages
Tesnet sync failed with:
Feb 07 00:04:19 nodoka neo-go[1747]: 2020-02-07T00:04:19.838+0300        WARN        blockQueue: failed adding block into the blockchain        {"error": "failed to store notifications: not supported", "blockHeight": 713984, "nextIndex": 713985}

because some (not so) smart contract emitted a notification with an
InteropItem inside.
2020-02-07 15:34:59 +03:00
Evgenii Stratonikov
8243a8b3a7 emit: use io.BinWriter instead of bytes.Buffer 2020-02-06 18:45:37 +03:00
Evgenii Stratonikov
4d8a3a359b vm: move Emit* functions to a separate package
Also strip 'Emit' prefix because 'emit' is now
in the package name.
2020-02-06 18:45:37 +03:00
Roman Khimov
ab14a4619d
Merge pull request #647 from nspcc-dev/fix-mempool-and-chain-locking
Fix mempool and chain locking

This allows us easily make 1000 Tx/s in 4-nodes privnet, fixes potential
double spends and improves mempool testing coverage.
2020-02-06 18:43:29 +03:00
Evgenii Stratonikov
9204b95d5b core: fix GAS price definitions 2020-02-06 18:19:08 +03:00
Roman Khimov
18695e660b mempool: drop RemoveOverCapacity(), handle it right in the Add()
Simplifies things a lot and removes useless code.
2020-02-06 17:50:11 +03:00
Roman Khimov
684cbf5bac mempool: make it almost 100% test-covered 2020-02-06 17:50:11 +03:00
Roman Khimov
6672f4b26f mempool: iterate over slice in GetVerifiedTransactions()
It's more efficient and keeps transactions sorted by priority.
2020-02-06 16:59:20 +03:00
Roman Khimov
e97396e56c mempool: test addition of conflicting tx to the pool 2020-02-06 15:57:20 +03:00
Roman Khimov
1133bbe584 mempool: remove unverified transactions pool
Our mempool only contains valid verified transactions all the time, it never
has any unverified ones. Unverified pool made some sense for quick unverifying
after the new block acceptance (and gradual background reverification), but
reverification needs some non-trivial locking between blockchain and mempool
and internal mempool state locking (reverifying tx and moving it between
unverified and verified pools must be atomic). But our current reverification
is fast enough (and has all the appropriate locks), so bothering with
unverified pool makes little sense.
2020-02-06 15:45:16 +03:00
Roman Khimov
b675903f52 mempool/core: redesign mempool dances on block acceptance
We not only need to remove transactions stored in the block, but also
invalidate some potential double spends caused by these transactions. Usually
new block contains a substantial number of transactions from the pool, so it's
easier to make one pass over it only keeping valid items rather than remove
them one by one and make an additional pass to recheck inputs/witnesses.
2020-02-06 15:45:16 +03:00
Roman Khimov
b567ce86ac mempool: implement insertion to sorted slice
Which is way faster than sort.Sort'ing things all the time.
2020-02-06 15:44:18 +03:00
Roman Khimov
35183b6dba mempool: reverse the order of sorted slice
Chopping off the last element of the slice if way easier than doing it with
the first one.
2020-02-06 15:44:16 +03:00
Roman Khimov
794027a90b mempool: use one slice for both priorities
It doesn't harm as we have transactions naturally ordered by fee anyway and it
makes managing them a little easier. This also makes slices store item itself
instead of pointers to it which reduces the pressure on the memory subsystem.
2020-02-06 15:43:45 +03:00
Roman Khimov
325bea3fa9 mempool: cache Feer invocation results in the item
They shouldn't depend on the chain state and for the same transaction they
should always produce the same result. Thus, it makes no sense recalculating
them over and over again.
2020-02-06 15:41:56 +03:00
Roman Khimov
e01bfeeb4d mempool: remove lock indirection from the Pool
After the f0bb886be3 with all methods of Pool
being pointer-based it makes no sense having this lock as a pointer.
2020-02-06 15:41:56 +03:00
Roman Khimov
a928ad9cfa mempool: make item an internal thing of mempool package
Nobody outside should care about these details, mempool operates on
transactions and that's it.
2020-02-06 15:41:56 +03:00
Roman Khimov
f0e3a31bc8 mempool: fix appending to sorted pools
Appending and not changing the real Items is utterly wrong.
2020-02-06 15:41:56 +03:00
Roman Khimov
f9963cca37 core: short-circuit verifyInputs when there are no inputs 2020-02-06 15:41:56 +03:00
Roman Khimov
70b3839fd0 core/mempool: fix AddBlock and tx pooling concurrency issues
Eliminate races between tx checks and adding them to the mempool, ensure the
chain doesn't change while we're working with the new tx. Ensure only one
block addition attempt could be in progress.
2020-02-06 15:41:52 +03:00
Roman Khimov
f0bb886be3 mempool: make all methods pointer methods
Makes no sense copying the Pool around.
2020-02-04 17:36:11 +03:00
Roman Khimov
388fed06e5 transaction: forbid serializing invalid transactions
Transaction that has no data is invalid and can't be serialized, so throw an
error if someone tries to.
2020-02-03 16:33:03 +03:00
Evgeniy Kulikov
eb59460032
Merge pull request #614 from nspcc-dev/configurable-mempool-size
config: add MemPoolSize configuration parameter
2020-01-23 13:25:27 +03:00
Roman Khimov
d6784e8f45 config: add MemPoolSize configuration parameter
Allow tuning/experimenting with mempool.
2020-01-23 13:10:44 +03:00
Evgenii Stratonikov
303b82cc44 core: set free GAS limit via config 2020-01-22 14:32:21 +03:00
Evgenii Stratonikov
4718375db1 rpc: return amount of consumed GAS in invoke* 2020-01-22 14:31:57 +03:00
Evgenii Stratonikov
4eae55143f core: restrict GAS available for invocation transactions
There are 10 GAS available for free plus any amount of GAS
attached to a transaction.
2020-01-22 14:31:54 +03:00
Roman Khimov
32213b1454
Merge pull request #601 from nspcc-dev/refactoring/core
core: refactor out Block, BlockBase and Header, closes #597.
2020-01-20 16:19:20 +03:00
Roman Khimov
46b82b4fb5 keys: don't return error from PrivateKey.Sign
As it can't ever happen.
2020-01-17 17:00:30 +03:00
Evgenii Stratonikov
28183b81d6 mempool: simplify names of exported types
With the move to a separate package, naming can be simplified:
MemPool -> Pool, PoolItem -> Item, PoolItems -> Items.
2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
fed6fba9b6 core: refactor out MemPool 2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
79bceb3e40 block: use require in tests 2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
489b88afbb block: rename BlockBase to Base 2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
63c56cca5c core: refactor out Block, BlockBase and Header structs
See #597.
2020-01-16 10:16:24 +03:00
Evgenii Stratonikov
9dc5571327 core,rpc: close Blockchain in tests
If blockchain is not closed, logging in defer can occur
after test has finished, which will lead to a panic with
"Log in goroutine after Test* has completed".
2020-01-10 11:47:56 +03:00
Evgenii Stratonikov
637c99eda7 core: simplify newInteropContext 2020-01-10 11:14:29 +03:00
Evgenii Stratonikov
aecdf470e7 cli,pkg: use zap.Logger 2020-01-10 11:14:27 +03:00
Evgenii Stratonikov
9c79684516 storage: panic on error in boltdb.Seek
Error in Seek means something is terribly wrong (e.g. db was not opened) and
error drop is not the right thing to do, because caller
will continue working with the wrong view.
2020-01-09 15:17:35 +03:00
Roman Khimov
c306d2f07c
Merge pull request #581 from nspcc-dev/feat/storage_find
core: implement Neo.Storage.Find interop, closes #422.
2019-12-26 15:46:46 +03:00
Evgenii Stratonikov
48cf4f4a84 core: implement Neo.Storage.Find interop 2019-12-26 15:21:41 +03:00
Roman Khimov
f8887b63ba
Merge pull request #578 from nspcc-dev/fix/sign
consensus: verify payloads correctly, closes #555.
2019-12-26 13:07:40 +03:00
Evgenii Stratonikov
714c466c2c core: add ScriptFromWitness function
Extracting verification script from witness became a common task.
This commit adds such a possibility.
2019-12-26 10:49:56 +03:00
Roman Khimov
9145855d2c
Merge pull request #579 from nspcc-dev/refactor-crypto
This moves some functionality into micro-packages, improves testing,
unexports some code and fixes bugs along the way.
2019-12-25 18:12:50 +03:00
Roman Khimov
b246653f62 address: rename functions as per #579 comments
Make them more clear to understand.
2019-12-25 17:34:18 +03:00
Roman Khimov
e685e9bf9a address: move into its own package
Doesn't really belong to the crypto.
2019-12-25 15:22:02 +03:00
Roman Khimov
ee28fb08f6 crypto: move merkle tree into the hash package
It's all about hashes, so it makes sense putting it there.
2019-12-25 11:28:59 +03:00
Vsevolod Brekelov
7f1844a39e transaction: add unmarshaling for TXType 2019-12-25 01:44:37 +03:00
Evgenii Stratonikov
5bc32b523a vm: implement Neo.Iterator.* interops 2019-12-24 10:21:47 +03:00
Evgenii Stratonikov
3ff7fd5262 vm: implement Neo.Enumerator.* interops 2019-12-24 10:21:46 +03:00
Roman Khimov
bf84e1f2fb core: cache top block
Turns out, our dApps use it a lot and we were going to the DB to get it which
is a useless waste of time. Technically we could also remove blockHeight here,
but not doing it at the moment as it's more involved.

It eliminates this time waste from the pprof graph, but doesn't change 1.4M ->
1.5M 100K mainnet block import test case in any noticeable way.
2019-12-23 19:18:12 +03:00
Roman Khimov
deb76c5199 core: add negative non-interop test for all interops
Try to feed some garbage into these functions.
2019-12-23 16:14:43 +03:00
Roman Khimov
76d69ee5ee core: restore proper interops ordering 2019-12-23 16:14:43 +03:00
Roman Khimov
c287a9e93c core: implement Neo.Witness.GetVerificationScript interop 2019-12-23 16:14:43 +03:00
Roman Khimov
9d638d0fee core: implement Neo.InvocationTransaction.GetScript 2019-12-23 16:14:43 +03:00
Roman Khimov
a7457d08a1 vm/core: add ID support for SYSCALL, redo interop registration
This solves two problems:
 * adds support for shortened SYSCALL form that uses IDs (similar to #434, but
   for NEO 2.0, supporting both forms), which is important for compatibility
   with C# node and mainnet chain that uses it from some height
 * reworks interop plugging to use callbacks rather than appending to the map,
   these map mangling functions are clearly visible in the VM profiling
   statistics and we want spawning a VM to be fast, so it makes sense
   optimizing it. This change moves most of the work to the init() phase
   making VM setup cheaper.

Caveats:
 * InteropNameToID accepts `[]byte` because that's the thing we have in
   SYSCALL processing and that's the most often usecase for it, it leads to
   some conversions in other places but that's acceptable because those are
   either tests or init()
 * three getInterop functions are: `getDefaultVMInterop`, `getSystemInterop`
   and `getNeoInterop`

Our 100K (1.4M->1.5M) block import time improves by ~4% with this change.
2019-12-19 13:35:42 +03:00
Evgenii Stratonikov
884779e501 *: goimports 2019-12-17 14:51:28 +03:00
Roman Khimov
7a5e995030 core: add MemPool Verify() test 2019-12-14 00:05:45 +03:00
Roman Khimov
5d68f88196 core: fix and speed up mempool Verify()
First of all, it was wrong, it was not checking for inputs really, it compared
tx hashes for some reason, second, when it did compare inputs it compared only
the PrevIndex part of them which is also wrong.

Also, there is absolutely no reason to go through GetVerifiedTransactions()
here, we don't need this copy of pointers and it can also be outdated by the
time we're to finish our check.

Before:
BenchmarkTXPerformanceTest-4
    5000            485506 ns/op           65886 B/op        409 allocs/op
ok      github.com/CityOfZion/neo-go/integration        3.212s

After:
enchmarkTXPerformanceTest-4
    5000            371104 ns/op           44367 B/op        408 allocs/op
ok      github.com/CityOfZion/neo-go/integration        2.712s
2019-12-13 23:23:33 +03:00
Roman Khimov
e631e75718 core: optimize GetVerifiedTransactions()
This simple change improves our BenchmarkTXPerformanceTest by 14%, just
because we don't waste time on reallocations during append().

Before:
   10000            439754 ns/op          218859 B/op        428 allocs/op
ok      github.com/CityOfZion/neo-go/integration        5.423s

After:
   10000            369833 ns/op           87209 B/op        412 allocs/op
ok      github.com/CityOfZion/neo-go/integration        4.612s
2019-12-13 19:57:17 +03:00
Roman Khimov
a9401e2ec7 core: do not write new dao elements into DB
It's useless work being done before it's actually needed. These (updated with
new values) are going to be written with some kind of Put anyway, so writing
them here is just a waste of time.
2019-12-13 19:57:17 +03:00
Roman Khimov
6896b40dee state: use open-coded array (de)serialization for accounts
We're spending a lot of time here, 100K blocks import starting at 1.4M, before
this patch:
real    4m17,748s
user    6m23,316s
sys     0m37,866s

After:
real    3m54,968s
user    5m56,547s
sys     0m39,398s

9% is quite a substantial improvement to justify this change.
2019-12-13 19:57:17 +03:00
Roman Khimov
0d0a27d271 core: add cachedDao to cache accounts and contracts
Importing 100K blocks starting at 1.4M, before this patch:
real    6m0,356s
user    8m52,293s
sys     0m47,372s

After this patch:
real    4m17,748s
user    6m23,316s
sys     0m37,866s

Almost 30% better.
2019-12-13 19:57:17 +03:00
Roman Khimov
212cf44e26 core: get data from dao in interops
It's more logically correct as Blockchain can be several transactions behind
in terms of its state.
2019-12-13 17:17:14 +03:00
Roman Khimov
0abd55c2c2 vm: add cached ScriptHash() to Context
Avoid recalculating it over and over again in interop.
2019-12-13 17:05:03 +03:00
Roman Khimov
09223236f9 core: add Persist() to dao()
Hide its internals a little, which is gonna be useful for future composition.
2019-12-12 21:17:13 +03:00
Roman Khimov
eb1749d190 core: remove persisting from GetValidators()
It's a getter function and even though it's quite fancy with its transactions
processing (for consensus operation) it shouldn't ever change the state of the
Blockchain. If we're to change anything here these changes may conflict with
the actual block processing later or may lead to broken state (if transactions
won't be approved for some reason).
2019-12-12 21:14:33 +03:00
Roman Khimov
31dbc35279 core: gofmt dao_test.go 2019-12-12 21:05:36 +03:00
Roman Khimov
c5ed3b788b core: simplify dao creation with newDao() 2019-12-12 21:04:55 +03:00
Roman Khimov
8b3080b972 io: rename Read/WriteBytes to Read/WriteB
go vet is not happy about them:
  pkg/io/binaryReader.go:92:21: method ReadByte() byte should have signature ReadByte() (byte, error)
  pkg/io/binaryWriter.go:75:21: method WriteByte(u8 byte) should have signature WriteByte(byte) error
2019-12-12 20:19:50 +03:00
Roman Khimov
54d888ba70 io: add type-specific read/write methods
This seriously improves the serialization/deserialization performance for
several reasons:
 * no time spent in `binary` reflection
 * no memory allocations being made on every read/write
 * uses fast ReadBytes everywhere it's appropriate

It also makes Fixed8 Serializable just for convenience.
2019-12-12 20:19:50 +03:00
Roman Khimov
89d7f6d26e core/tx: microoptimize block/tx hashing
Don't hash the data twice.
2019-12-12 18:01:30 +03:00
Vsevolod Brekelov
c93a8d2bc4 core: extracted same logic to separate methods 2019-12-11 13:14:45 +03:00
Vsevolod Brekelov
c1f39d5c7b internal: moved testutil method to internal package 2019-12-11 13:14:43 +03:00
Vsevolod Brekelov
2d42b14a1d core: renames entities-> state and removed State prefix 2019-12-11 13:14:18 +03:00
Vsevolod Brekelov
8809fe437d core: unit tests for dao 2019-12-11 13:13:51 +03:00
Vsevolod Brekelov
c0e59ebd4e core: unit tests for entities 2019-12-11 13:05:31 +03:00
Vsevolod Brekelov
c7ac4b6dff core: fix encoding and decoding for notification event 2019-12-11 13:05:31 +03:00
Vsevolod Brekelov
ec17654986 core: refactoring blockchain state and storage
add dao which takes care about all CRUD operations on storage
remove blockchain state since everything is stored on change
remove storage operations from structs(entities)
move structs to entities package
2019-12-11 13:05:31 +03:00
Roman Khimov
d0f9a28196 vm/core: improve block import speed with PublicKey caching
This change (closely related to the neo-project/neo#1321 proposal) speeds up
1.4M mainnet blocks import by 30%. Basically, we're eliminating key decoding
for block's multisignature that has the same keys most of the time.

Things I don't like about this patch:
 * yet another parameter for verifyHashAgainstScript()
 * vm keys are not copied in/out

But it's rather simple and solves the problem for this particular case, so I
think it's worth it.
2019-12-10 19:13:29 +03:00
Roman Khimov
f1856bfa8b core/tx: remove publickey indirection from assets and txes
It makes very little sense having pointers here, these structures MUST have
some kind of key and this key is not gonna be wandering somewhere on its
own. Fixes a part of #519.
2019-12-09 18:33:04 +03:00
Roman Khimov
5b6c5af704 *: implement EncodeBinary with pointer receivers where appropriate
Everywhere except ParamType (which is just a byte), reduce copying things
around for no real reason.
2019-12-09 18:25:15 +03:00
Roman Khimov
7e371588a7 core/tx: remove one layer of indirection for scripts and inouts
It reduces heap pressure a little for these elements as we don't have to
allocate/free them individually. And they're directly tied to transactions or
block, not being shared or anything like that, so it makes little sense for
them to be pointer-based. It only makes building transactions a little easier,
but that's obviously a minor usecase.
2019-12-09 17:14:10 +03:00
Evgenii Stratonikov
fccb008594 io: implement ReadBytes() 2019-12-09 15:00:15 +03:00
Evgenii Stratonikov
838050f8b5 io: rename ReadBytes() to ReadVarBytes() 2019-12-09 15:00:15 +03:00
Roman Khimov
e4d821f32d
Merge pull request #546 from nspcc-dev/write-optimizations
Write optimizations
2019-12-06 19:40:38 +03:00
Roman Khimov
9992a98007 core: optimize balance utxo removal
One of my samples had 8 (out of 30) seconds spent here, but values of 100ms
are more typical. After this change it becomes invisible.
2019-12-06 18:46:24 +03:00
Roman Khimov
844491d365 *: use more efficient WriteBytes where appropriate
Before this patch on block import we could easily be spending more than 6
seconds out of 30 in Uint256 encoding for UnspentBalance, now it's completely
off the radar.
2019-12-06 18:22:21 +03:00
Evgenii Stratonikov
7179e4ba9f util: add LE suffix to Uint256 methods 2019-12-06 12:16:55 +03:00
Evgenii Stratonikov
57efad912c util: add LE suffix to Uint160 methods 2019-12-06 12:16:55 +03:00
Roman Khimov
8beb135829
Merge pull request #542 from nspcc-dev/fix-storage-interop-reads
core: fix wrong data being read in interops (part of #501)
2019-12-05 09:08:38 +03:00
Roman Khimov
c034aae378 core: fix wrong data being read in interops (part of #501)
When 74590551 introduced this code we had no proper caching layer, so there
were these strange fallbacks in the code. fc0031e5 should'd removed them, but
failed to do so, so do it now and fix processing of transactions that touch
storage for the same key (address) in the same block.
2019-12-04 19:51:57 +03:00
Vsevolod Brekelov
ff15a0acfd core: add trigger types as constants. Closes #509 2019-12-04 12:27:04 +03:00
Roman Khimov
31add423a8 core: replace open-coded PUSHBYTES64 with the one from opcode 2019-12-03 18:23:46 +03:00
Roman Khimov
138e125646 *: remove duplicate functions producing verification script
Drop wif.GetVerificationScript(), drop
smartcontract.CreateSignatureRedeemScript(), add GetVerificationScript()
directly to the PublicKey and use it everywhere.
2019-12-03 18:23:46 +03:00
Roman Khimov
8d4dd2d2e1 vm: move opcodes into their own package
This allows easier reuse of opcodes and in some cases allows to eliminate
dependencies on the whole vm package, like in compiler that only needs opcodes
and doesn't care about VM for any other purpose.

And yes, they're opcodes because an instruction is a whole thing with
operands, that's what context.Next() returns.
2019-12-03 18:22:14 +03:00
Roman Khimov
f48228ef7d
Merge pull request #467 from nspcc-dev/errcheck_297
This patchset closes #297 and #457.
2019-12-03 15:06:11 +03:00
Vsevolod Brekelov
03ff2976ed io: refactoring for using WriteVarBytes instead of WriteLE
goal is to be consistent with C# implementation.
For writing []byte WriteBytes used and for byte - WriteVarByte.
2019-12-03 13:49:33 +03:00
Roman Khimov
1c89c192ac core: optimize some accesses to unsortedTxn in mempool
These don't need a full lock, they only read things from maps.
2019-12-02 22:39:43 +03:00
Roman Khimov
0cd3493fa5 core: fix potential locking problem in mempool
I think it should fix this issue mentioned in the #526:

INFO[1456] blockchain persist completed                  blockHeight=63480 headerHeight=1810000 persistedBlocks=1 persistedKeys=4 took=740.7113ms
fatal error: concurrent map read and map write

goroutine 322 [running]:
runtime.throw(0xc8a6dc, 0x21)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc003473958 sp=0xc003473928 pc=0x42e282
runtime.mapaccess2(0xb706a0, 0xc0001893b0, 0xc0034739c8, 0xc0028704e0, 0x3)
        /usr/local/go/src/runtime/map.go:470 +0x278 fp=0xc0034739a0 sp=0xc003473958 pc=0x40dc08
github.com/CityOfZion/neo-go/pkg/core.MemPool.ContainsKey(0xc0001d0d20, 0xc0001893b0, 0xc0001893e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /neo-go/pkg/core/mem_pool.go:92 +0xcb fp=0xc003473a30 sp=0xc0034739a0 pc=0x9326db
github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).HasTransaction(0xc0001ca8c0, 0x49f0b45d430e441b, 0x553db79b7072821c, 0x28969518de11976, 0xba5100efddbe79d4, 0xc003473cd0)
        /neo-go/pkg/core/blockchain.go:803 +0xa1 fp=0xc003473b68 sp=0xc003473a30 pc=0x914b11
github.com/CityOfZion/neo-go/pkg/core.Blockchainer.HasTransaction-fm(0x49f0b45d430e441b, 0x553db79b7072821c, 0x28969518de11976, 0xba5100efddbe79d4, 0xc005a5d388)
        /neo-go/pkg/core/blockchainer.go:28 +0x46 fp=0xc003473ba8 sp=0xc003473b68 pc=0x997326
github.com/CityOfZion/neo-go/pkg/network.(*Server).handleInvCmd(0xc00018f680, 0xd9d980, 0xc00025e190, 0xc005a5d380, 0x0, 0x0)
        /neo-go/pkg/network/server.go:401 +0x3bb fp=0xc003473d38 sp=0xc003473ba8 pc=0x9924cb
github.com/CityOfZion/neo-go/pkg/network.(*Server).handleMessage(0xc00018f680, 0xd9d980, 0xc00025e190, 0xc007a0d050, 0x0, 0x0)
        /neo-go/pkg/network/server.go:582 +0x1ae fp=0xc003473da0 sp=0xc003473d38 pc=0x993bbe
github.com/CityOfZion/neo-go/pkg/network.(*TCPTransport).handleConn(0xc000228420, 0xd9b880, 0xc0001b6f00)
        /neo-go/pkg/network/tcp_transport.go:93 +0x202 fp=0xc003473fc8 sp=0xc003473da0 pc=0x996672
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc003473fd0 sp=0xc003473fc8 pc=0x45b3e1
created by github.com/CityOfZion/neo-go/pkg/network.(*TCPTransport).Dial
        /neo-go/pkg/network/tcp_transport.go:36 +0xb4

The problem is that we're modifying `unsortedTxn` under a reader lock.
2019-12-02 22:36:59 +03:00
Roman Khimov
bc5beb438f *: gofmt
Somehow these crept into the repository.
2019-12-02 10:10:50 +03:00
Roman Khimov
2c0ed62680 core: fix blockchain init with (2000*N + 1) header hashes in DB
Stored header count includes block number zero, so it's (height + 1).
2019-11-29 16:26:44 +03:00
Roman Khimov
2d41450ac9
Merge pull request #507 from nspcc-dev/dbft
network: plug in dBFT library
2019-11-27 15:22:40 +03:00
Roman Khimov
c5e2b3ce73
Merge pull request #523 from nspcc-dev/fix-interim-block-changes-store
Fix interim block changes store
2019-11-27 14:51:29 +03:00
Vsevolod
1a9e98caf3
Merge pull request #522 from nspcc-dev/fix-invalid-tx-type-handling
Fix invalid tx type handling
2019-11-27 13:40:52 +03:00
Roman Khimov
467c9c146d core: restore intermediate block-level store
Commit c80ee952a1 removed temporary store used
to contain changes of the block being processed. It's wrong in that the block
changes should be applied to the database in a single transaction so that
there wouldn't be any intermediate state observed from the outside (which is
possible now). Also, this made changes commiting persist them to the
underlying store effectively making our persist loop a no-op (and not
producing `persist completed` log lines that we love so much).
2019-11-27 13:28:13 +03:00
Roman Khimov
bb765ccfa7 core: gofmt 2019-11-27 13:27:47 +03:00
Roman Khimov
65bf0d01d5 transaction: don't hide decoding error by logging
Unknown transaction types are wrong and should be treated as errors.
2019-11-27 13:12:04 +03:00
Evgenii Stratonikov
fdd5276d3e network: plug in dBFT library 2019-11-27 10:57:22 +03:00
Evgenii Stratonikov
af61571ba0 core: do not return error in block.getHashableData 2019-11-26 15:27:37 +03:00
Roman Khimov
eb84ae49da
Merge pull request #508 from nspcc-dev/feature/getvalidators_420
core: Implement getvalidators interop
2019-11-22 18:15:08 +03:00
Vsevolod Brekelov
c03de88b41 core: unit tests blockchain state 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
d28f2dce05 core: unit test validator state 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
419b031e96 core: add weighted function for validators 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
672f314be0 core: nit refactoring and add godoc 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
8ae88593dd core: blockchain fmt 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
3c4b0e955d core: create blockchain state that represents state of the bc
move some util methods to blockchain state
2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
c80ee952a1 core/crypto: add validators and interop for validators
add processing of validators while block persist;
add validator structure with decoding/encoding;
add validator get from store;
add EnrollmentTX and StateTX processing;
add pubkey decode bytes, unique and contains functions;
2019-11-21 15:23:35 +03:00
Roman Khimov
34e2122e58 core: only check tx against mempool if it's not in the block
Fixes failure to process transaction from the block when it was relayed
initially:

WARN[0788] blockQueue: failed adding block into the blockchain  blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
WARN[0790] blockQueue: failed adding block into the blockchain  blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
WARN[0790] blockQueue: failed adding block into the blockchain  blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
2019-11-20 16:12:40 +03:00
Roman Khimov
27a57e1a2d transaction: allow system fee specification in NewInvocationTX
It's not possible to create any deployment TX without it.
2019-11-20 16:12:40 +03:00
Roman Khimov
29882b076c rpc: remove duplicating definition of UTXO
Port sorting methods to core.
2019-11-19 17:35:04 +03:00
Roman Khimov
d2bdae99e4
Merge pull request #492 from nspcc-dev/feature/interop_neo_tests1
core: add unit tests for interops_neo
2019-11-18 19:39:05 +03:00
Vsevolod Brekelov
d57b6e1091 core: add unit tests for interops_neo 2019-11-18 19:31:27 +03:00
Roman Khimov
5841d3931e rpc: implement getunspents method, fix #473 2019-11-18 11:30:09 +03:00
Roman Khimov
e4868cd429 core: implement EncodeBinary on value
Fixes panic:
panic: core.NotificationEvent does not have EncodeBinary(*BinWriter)

goroutine 97 [running]:
github.com/CityOfZion/neo-go/pkg/io.(*BinWriter).WriteArray(0xc004f7dda0, 0xabf820, 0xc004f7ddc0)
        /home/rik/dev/neo-go/pkg/io/binaryWriter.go:45 +0x4b7
github.com/CityOfZion/neo-go/pkg/core.(*AppExecResult).EncodeBinary(0xc000635498, 0xc004f7dda0)
        /home/rik/dev/neo-go/pkg/core/notification_event.go:73 +0x9c
github.com/CityOfZion/neo-go/pkg/core.putAppExecResultIntoStore(0xcf0020, 0xc001aafec0, 0xc000635498, 0xc004f81940, 0x40)
        /home/rik/dev/neo-go/pkg/core/notification_event.go:33 +0xab
github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).storeBlock(0xc000118000, 0xc006f89ea0, 0xc006f89ea0, 0x0)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:534 +0x2539
github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).AddBlock(0xc000118000, 0xc006f89ea0, 0xc004f368b0, 0x1)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:256 +0x91
github.com/CityOfZion/neo-go/pkg/network.(*blockQueue).run(0xc0048f4b80)
        /home/rik/dev/neo-go/pkg/network/blockqueue.go:39 +0x14c
created by github.com/CityOfZion/neo-go/pkg/network.(*Server).Start
        /home/rik/dev/neo-go/pkg/network/server.go:133 +0x272
2019-11-15 22:05:09 +03:00
Roman Khimov
1a5731e005 core: rework balance accounting
Store all unspents instead of balance values. Bump store version as this is an
incompatible change.
2019-11-15 22:05:09 +03:00
Roman Khimov
a16c2c3825 core: add NEP5 transfer tracking stub
See #498.
2019-11-15 18:04:59 +03:00
Roman Khimov
01082a8988 core: fix nondeterministic txGetReferences() behavior
Ranging over map is nondeterministic and contracts may be unprepared for
that. Fixes #454.
2019-11-15 18:04:59 +03:00
Roman Khimov
5e8122bfac core: add contract hash into the error output for checkStorageContext()
Makes debugging things easier.
2019-11-15 18:04:59 +03:00
Roman Khimov
5ce269c035 core: change notify implementation to save notifications
Fixes #453 and makes it possible to refer to these notifications later.
2019-11-15 18:04:59 +03:00
Evgenii Stratonikov
1a08ad19aa sc: implement io.Serializable for ParamType 2019-11-14 14:20:38 +03:00
Evgenii Stratonikov
2be18f91df util: implement io.Serializable for Uint256 2019-11-14 14:20:38 +03:00
Evgenii Stratonikov
b16e56a47b io: refactor BinReader.ReadArray()
Make it accept arbitrary slice pointer.
2019-11-14 14:19:58 +03:00
Evgenii Stratonikov
ad9091d13d io: implement generic array (de-)serialization
It is done through reflection and panics
in every unexpected situation.
2019-11-13 17:27:23 +03:00
Roman Khimov
394346fc26 core: add proper claim transactions processing 2019-11-11 15:35:57 +03:00
Roman Khimov
b05754deac core: add Close() to blockchainer, implement it to properly close chain
Before it the deferred function in Run() was actually never able to properly
close the Store, so we weren't synching the latest state to the disk.
2019-11-08 12:19:54 +03:00
Vsevolod
f686069f37
Merge pull request #481 from nspcc-dev/core-fix-init-corner-cases
Fix init corner cases
2019-11-06 18:36:40 +03:00
Roman Khimov
c567307748 core: fix potential issue with header list init
If we're to receive some 500 headers (less than `headerBatchCount`) and quit
before receiving more of them we end up with clean `bc.headerList` that will
be inited going backwards to the `targetHash`, but code path doesn't add add
the `targetHash` itself which it should do in this particular case, otherwise
we end with no genesis block hash in the list.
2019-11-06 18:13:45 +03:00
Roman Khimov
b5199625cd core: init genesis block header if there are no other headers in DB
Fixes crash when restarted after the DB initialization and no blocks written
into the DB.
2019-11-06 17:58:19 +03:00
Roman Khimov
c16c2bf102 core: store current header reference on clean init
Missing it the following line could fail on subsequent restarts:

        currHeaderHeight, currHeaderHash, err := storage.CurrentHeaderHeight(bc.store)

if the node was stopped before any headers had been received.
2019-11-06 17:56:06 +03:00
Vsevolod Brekelov
d799c98cfe io: move common function and add unit tests for it 2019-11-06 17:12:33 +03:00
Roman Khimov
9f7625d699 core: don't do useless DB reads if nothing was persisted 2019-11-06 16:10:37 +03:00
Roman Khimov
90d4299f0b core: gofmt blockchain.go 2019-11-06 16:09:45 +03:00
ed758458d5 vm: implement serialization interops in core package 2019-11-05 17:10:52 +03:00
Roman Khimov
2f6e678a19
Merge pull request #463 from nspcc-dev/smartcontract-fixes
Smartcontract RPC fixes
2019-10-29 20:54:46 +03:00
Vsevolod Brekelov
d374175170 monitoring: add prometheus monitoring
add init metrics service which uses prometheus;
add configuration for metrics service;
add monitoring metrics for blockchain,rpc,server;
2019-10-29 20:51:17 +03:00
Roman Khimov
ebc1ba4f38 rpc/core: implement invokescript method, fix #348
Extend Blockchainer with one more method to spawn a VM for test runs and use
it to run scripts. Gas consumption is not counted or limited in any way at the
moment (see #424).
2019-10-29 18:31:39 +03:00
Roman Khimov
a568740f56 core: deduplicate a part of VM setup into spawnVMWithInterops() 2019-10-29 17:10:50 +03:00
Roman Khimov
99bbad331d core: add issue tx processing
Count available asset quantity, match C# code.
2019-10-25 17:20:12 +03:00
Roman Khimov
45cac07643 core: remove transactions from MemPool when they're stored in block
Extend MemPool with Remove() and use it. Fixes #446.
2019-10-24 13:12:02 +03:00
Roman Khimov
184d8a0180 core: fix missing unlock in MemPool TryAdd()
Fixes deadlock on subsequent MemPool operations.
2019-10-24 13:12:01 +03:00
Roman Khimov
8d0ed9259c transaction: fix invocation TX consistency in NewInvocationTX()
NewInvocationTX() returned a version number one transaction that actually
failed to pass that version down to the invocation data which lead to
serialization/deserialization inconsistency.
2019-10-23 14:50:24 +03:00
Roman Khimov
1250bf9579
Merge pull request #443 from nspcc-dev/spellcheck
spellcheck and comments fix
2019-10-22 18:01:40 +03:00
Vsevolod Brekelov
8ee421db14 fix spelling and godoc comments 2019-10-22 17:56:03 +03:00
Roman Khimov
3cbb699eb7
Merge pull request #426 from nspcc-dev/logger_247
Change fmt.Println to log, close #247.
2019-10-22 14:41:30 +03:00
Vsevolod Brekelov
e2bfff8666 vm: removed mute mode and pushed logging to upper lvl
VM should be responsible for code execution and in case anyone interested in additional logging or errors they could handle them like we do it iin cli.
2019-10-22 13:44:14 +03:00
Roman Khimov
f1750d117b
Merge pull request #440 from nspcc-dev/db-dump-restore
DB dump and restore, fixes #436.
2019-10-21 15:42:05 +03:00
Roman Khimov
07c2105aa5 core: log values from the store in persist()
We're about stored values here, so print those, which avoids blocking in
bc.HeaderHeight() and removes duplication between blockHeight and
persistedHeight. Fixes saving the blockchain on exit (deferred function in
Run() blocked in persist()).

Test modification was required because storeBlocks() doesn't actually save
headers and thus TestGetTransaction started to fail on persist().
2019-10-21 14:18:09 +03:00
Roman Khimov
70407f0c19 core: remove unused context parameter from persist() 2019-10-21 14:18:09 +03:00
Vsevolod Brekelov
063fe5be9d unspentcoinstate: fix branch never executed 2019-10-21 12:40:11 +03:00
Roman Khimov
b533dfceba core: don't panic on init when there are less than 2000 hashes
If you're to sync less than 2000 headers no batched header key-value is
gonna be written into the DB and init() would panic because
bc.headerList.Len() would return 0. Use genesis block as a target in this
case.
2019-10-21 08:37:01 +03:00
Vsevolod Brekelov
fe39c565b9 codestyle: redundant type conversion 2019-10-18 18:39:31 +03:00
Vsevolod Brekelov
1afdb895a1 codestyle: declare empty slice
https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices
2019-10-18 18:38:33 +03:00
Vsevolod Brekelov
ced2022273 storage: change println error for botldb to log 2019-10-17 18:23:49 +03:00
Roman Khimov
3fc2bf5452 *: fix some misspellings
Goreport:
   neo-go/pkg/core/contract_state_test.go
        Line 21: warning: "Contracto" is a misspelling of "Contraction" (misspell)
        Line 64: warning: "Contracto" is a misspelling of "Contraction" (misspell)

   neo-go/pkg/core/interop_neo.go
        Line 420: warning: "succeedes" is a misspelling of "succeeds" (misspell)

   neo-go/pkg/network/discovery.go
        Line 118: warning: "succeded" is a misspelling of "succeeded" (misspell)
        Line 128: warning: "successfuly" is a misspelling of "successfully" (misspell)
2019-10-17 12:30:24 +03:00
Roman Khimov
1b5ea67737 core: fix ineffassign suggestion
Goreport:
  Line 467: warning: ineffectual assignment to persisted (ineffassign)
2019-10-17 12:27:15 +03:00
Roman Khimov
288000a8af storage: drop useless nil assignment in leveldb code
golint suggests:
   Line 24: warning: should drop = nil from declaration of var opts; it is the zero value (golint)
2019-10-17 12:18:16 +03:00
Roman Khimov
2245fedbb1 core: deduplicate state commit methods, use interim MemCachedStore
commit methods duplicated putSmthIntoStore functions, but have MemCachedStore
now that can easily substitute for a Batch, especially given that interop
needs something like that for its storage purposes anyway.
2019-10-16 17:33:45 +03:00
Roman Khimov
fc0031e5aa core: move write caching layer into MemCacheStore
Simplify Blockchain and associated functions, deduplicate code, fix Get() and
Seek() implementations.
2019-10-16 17:33:45 +03:00
Roman Khimov
8926cbe368 core: fix potential data race in logging code
Spotted in CircleCI build with Go 1.12:

WARNING: DATA RACE
Write at 0x00c00011095c by goroutine 88:
  sync/atomic.SwapInt32()
      /usr/local/go/src/runtime/race_amd64.s:249 +0xb
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).persist()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:483 +0x172
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run.func2()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:191 +0x50

Previous read at 0x00c00011095c by goroutine 64:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).persist()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:492 +0x4be
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run.func2()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:191 +0x50

Goroutine 88 (running) created at:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:190 +0x264

Goroutine 64 (finished) created at:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:190 +0x264
2019-10-15 18:58:22 +03:00
Roman Khimov
a6610ba082 core: verify blocks, fix #12
This adds the following verifications:
 * merkleroot check
 * index check
 * timestamp check
 * witnesses verification

VerifyWitnesses is also renamed to verifyTxWitnesses here to not confuse it
with verifyBlockWitnesse and to hide it from external access (no users at the
moment).
2019-10-15 18:58:17 +03:00
Roman Khimov
03c20f1876 core: make ContractState receiver names consistent
Linter isn't happy with our recent changes:

pkg/core/contract_state.go:109:1: receiver name cs should be consistent with previous receiver name a for ContractState
pkg/core/contract_state.go:114:1: receiver name cs should be consistent with previous receiver name a for ContractState
pkg/core/contract_state.go:119:1: receiver name cs should be consistent with previous receiver name a for ContractState

But actually `a` here most probably is a copy-paste from AssetState methods,
so fit the old code to match the new one.
2019-10-15 12:56:25 +03:00
Roman Khimov
aec6a5f029 core/config: make block and transaction verification configurable
Enable transaction verification for privnets and tests, testnet can't
successfuly verify block number 316711 with it enabled and mainnet stops at
105829.
2019-10-15 12:56:25 +03:00
Roman Khimov
56dcff2894 core: invoke contracts for Invocation TXes
They can have some side-effects that future invocations rely on.
2019-10-15 12:56:25 +03:00
Roman Khimov
74590551c4 core: add some interops
This also changes Verify to VerifyTx and VerifyWitnesses, because there is a
need to pass a block for some interop functions.
2019-10-15 12:56:25 +03:00
Roman Khimov
667f346c04 core: improve error message in AddBlock test 2019-10-15 12:56:25 +03:00
Roman Khimov
8266a5ce19 core: export GetContractState/GetScriptHashesForVerifying via Blockchainer
These are gonna be used by interops and are also useful in general.
2019-10-15 12:56:25 +03:00
Roman Khimov
19fd7f844e core: add GetUnspentCoinState() for future interops 2019-10-15 12:56:25 +03:00
Roman Khimov
bfddf9b3f6 core: implement StorageItem for future interops 2019-10-15 12:56:25 +03:00
Roman Khimov
2156d5db73 core: add put/delete functions for ContractState
These are gonna be used by interops.
2019-10-15 12:56:25 +03:00
Roman Khimov
58d9b79fe3 core: avoid duplicates in GetScriptHashesForVerifyingClaim()
As they can break VerifyWitnesses().
2019-10-15 12:56:25 +03:00
Roman Khimov
35824728eb core: add putAssetStateIntoStore()
Will be used by interops.
2019-10-15 12:56:25 +03:00
Roman Khimov
96934cfeab core: return error from verifyOutputs()
Make debugging easier.
2019-10-15 12:56:25 +03:00
Roman Khimov
ab4ff79e02 core: fix typo in GetTransactionResults() 2019-10-15 12:56:25 +03:00
Roman Khimov
cffe8d0ee2 core: return error from verifyResults()
Make it easier to find out what the real problem is.
2019-10-15 12:56:25 +03:00
Roman Khimov
16bc5296cb block: return error from Verify
Don't hide real problem behind the bool value. Makes it easier to identify
problems when looking at log messages.
2019-10-15 12:56:25 +03:00
Roman Khimov
258f397b9a core: append transactions to the block in GetBlock()
We want to get a full block, so it has to have transactions
inside. Unfortunately our tests were used to this wrong behavior and utilized
completely bogus transactions without data that couldn't be persisted, so fix
that also.
2019-10-15 12:56:25 +03:00
Roman Khimov
238c590ddb core: fix contract state's Properties to use PropertyState
PublishTX only had one of these flags, but newer contracts (created via the
interop function) can have more and these flags are aggregated into one field
that uses PropertyState enumeration (it's used to publish contract, so
supposedly it's also a nice choice for contract state storage).
2019-10-15 12:56:25 +03:00
Roman Khimov
78861485b6 storage: simplify MemoryBatch
It's used a lot and it looks a lot like MemoryStore, it just needs not to
return errors from Put and Delete, so make it use MemoryStore internally with
adjusted interface.
2019-10-15 12:56:25 +03:00
Roman Khimov
13bf2618ef storage: improve PutBatch for MemoryStore
Make it look more like a real transaction, put/delete things with a single
lock. Make a copy of value in Put also, just for safety purposes, no one knows
how this value slice can be used after the Put.
2019-10-15 12:56:25 +03:00
Roman Khimov
3ada92944a storage: drop Len from the Batch interface
It's almost meaningless now and we can easily live without it.
2019-10-15 12:56:25 +03:00
Roman Khimov
e111892653 storage: redo DB testing
Make generic tests for all Store implementations, deduplicate tests. Implement
Delete() tests, test Seek() better, add LevelDB tests (finally!).
2019-10-15 12:56:25 +03:00
Roman Khimov
48b6a427cf storage: add Delete method for Batch and Store
It's gonna be used by Storage and Contract interops, both can delete their
data.
2019-10-07 19:40:11 +03:00
Roman Khimov
add9368e9d storage: use strings as keys for memory batch
Using pointers is just plain wrong here, because the batch can be updated with
newer values for the same keys.

Fixes Seek() to use HasPrefix also because this is the intended behavior.
2019-10-07 17:05:53 +03:00
Roman Khimov
1bf232ad50 vm: introduce TryBool() for Element and use it in VerifyWitnesses
Script can return non-bool results that can still be converted to bool
according to the usual VM rules. Unfortunately Bool() panics if this
conversion fails which is OK for things done in vm.execute(), but certainly
not for VerifyWitnesses(), thus there is a need for TryBool() that will just
return an error in this case.
2019-10-04 16:13:39 +03:00
Roman Khimov
ceca9cdb67 core/vm: implement contract storage and script retrieval
Fixes script invocations via the APPCALL instruction. Adjust contract state
field types accordingly.
2019-10-04 16:13:39 +03:00
Roman Khimov
b7c1efe7e3 core: fix References() result key type
If the block references two ouputs in some other transaction the code failed
to verify it because of key collision. C# code implements it properly by using
full CoinReference type as a key, so let's do it in a similar fashion.
2019-10-01 13:41:26 +03:00
Roman Khimov
8537700b7b core: sort hashes and witnesses in VerifyWitnesses()
Fixes failure to verify some multi-witnesses transactions in Testnet chain. C#
code contains similar logic.
2019-10-01 13:41:26 +03:00
Roman Khimov
dd0dd2d031 core: add claim-specific GetScriptHashesForVerifying()
Claim transactions have different logic in C# node, so we need to
implement it too. It's not the most elegant way to fix it, but let's make it
work first and then refactor if and where needed. Fixes verification of Claim
transactions.
2019-10-01 13:41:26 +03:00
Roman Khimov
854fb187a3 core: verify transactions in AddBlock() 2019-10-01 13:41:26 +03:00
Roman Khimov
1095abb04c core: fix default asset expiration to match C# code
Fixes bogus verification failures for non-native assets.
2019-10-01 13:41:26 +03:00
Roman Khimov
832c56f97d core: fix DutyFlag check in GetScriptHashesForVerifying()
It's a flag (used by Share and Invoice asset types), so it should be checked
like a flag, the same way C# node does.
2019-10-01 13:41:26 +03:00
Roman Khimov
4ae18e8ffc block: check for Transaction length before messing with txes
Fixes panic two lines below. Blocks without transactions are invalid by
definition, so there is a need to adjust tests accordingly.
2019-10-01 13:41:26 +03:00
Roman Khimov
7779ad6c28 storage: always return ErrKeyNotFound when key is not found
It was true for MemoryStore and BoltDB, but not for LevelDB and Redis.
2019-09-27 15:42:36 +03:00
Roman Khimov
5ff1399d54 storage: unexport MemoryStore mutex
It shouldn't be available from outside.
2019-09-27 15:42:36 +03:00
Roman Khimov
3c40a53c4d core: no need to Close() Blockchain in tests
It's a Store method actually and for every Blockchain that uses Run() the
Close() will be handled automatically on exit because of context magic.
2019-09-27 15:42:35 +03:00
Roman Khimov
0dbdbb9c2a storage: remove RedisBatch in favor of MemoryBatch
They're also almost the same.
2019-09-27 15:42:35 +03:00
Roman Khimov
af557cea19 storage: deduplicate BoltDBBatch/MemoryBatch
BoltDB doesn't have internal batching mechanism, thus we have a substitute for
it, but this substitute is absolutely identical to MemoryBatch, so it's better
to unify them and import ac5d2f94d3 fix into the
MemoryBatch.
2019-09-27 15:42:35 +03:00
Roman Khimov
8bf37f45fc storage: remove impossible error branch from MemoryStore
MemoryStore is always successful, so this branch makes no sense.
2019-09-27 15:42:35 +03:00
Roman Khimov
4a732d97ac storage: add comments about errors from MemoryStore
It never returns errors for some operations and this knowledge can be used by
other code.
2019-09-27 15:42:35 +03:00
Roman Khimov
920d7c610c core: remove blockCache, use MemoryStore, redesign persist()
Commit 578ac414d4 was wrong in that it saved
only a part of the block, so depending on how you use blockchain, you may
still see that the block was not really processed properly. To really fix it
this commit introduces intermediate storage layer in form of memStore, which
actually is a MemoryStore that supports full Store API (thus easily fitting
into the existing code) and one extension that allows it to flush its data to
some other Store.

It also changes AddBlock() semantics in that it only accepts now successive
blocks, but when it does it guarantees that they're properly added into the
Blockchain and can be referred to in any way. Pending block queing is now
moved into the server (see 8c0c055ac657813fe3ed10257bce199e9527d5ed).

So the only thing done with persist() now is just a move from memStore to
Store which probably should've always been the case (notice also that
previously headers and some other metadata was written into the Store
bypassing caching/batching mechanism thus leading to some inefficiency).
2019-09-27 15:42:35 +03:00
Roman Khimov
c531dc0bde network: add block queue
This one will replace blockCache in Blockchain itself as it can and should be
external from it. The idea is that we only feed successive blocks into the
Blockchain and it only stores valid proper Blockchain and nothing else.
2019-09-27 13:00:09 +03:00
Roman Khimov
903173c991 core: fix deadlock in headers list init
This probably is a problem since adc880d323,
because AddHeaders() uses headersOp channel and its reader is only ran in
Run().
2019-09-25 17:52:46 +03:00
Roman Khimov
578ac414d4 core: make unpersisted blocks/txs available in Blockchain
This changes the Blockchain to also return unpersisted (theoretically, verified
in the AddBlock!) blocks and transactions, making Add/Get interfaces
symmetrical. It allows to turn Persist into internal method again and makes it
possible to enable transaction check in GetBlock(), thus fixing #366.
2019-09-25 17:46:28 +03:00
Roman Khimov
8a457ef540 transaction: add a check for createHash() invocation
It should work in general.
2019-09-24 13:01:54 +03:00
Roman Khimov
679e01bd0f core: implement witness verification logic
Fix #368.
2019-09-24 13:01:54 +03:00
Roman Khimov
ac5d2f94d3 storage: fix BoltDB batched Put()
It must copy both the value and the key because they can be reused for other
purposes between Put() and PutBatch(). This actually happens with values in
headers processing, leading to wrong data being written into the DB.

Extend the batch test to check for that.
2019-09-23 09:27:18 +03:00
Roman Khimov
4b4bac675b core: don't print persisting errors twice
As they're already printed in the calling goroutine. And they're alse not
always useful when printed by Persist().
2019-09-22 20:08:15 +03:00
Roman Khimov
102c926ef3 core: don't print useless persist messages with nil errors
Errors should be printed only when there are errors.
2019-09-22 20:07:49 +03:00
Roman Khimov
5cddfe071b
Merge pull request #403 from nspcc-dev/fix_rpctest
Fix rpctest, fix #353, fix #305.
2019-09-18 23:14:37 +03:00
Vsevolod Brekelov
100fee164b unitTest: reworked RPC unit test
earlier we had an issue with failing test in #353 and other one #305.
Reworked these test to have in-memory database. This led to multiple
changes: made some functions like Hash and Persist public(otherwise
it's not possible to control state of the blockchain); removed
unit_tests storage package which was used mainly for leveldb in unit
tests.
I see these tests not really good since they look like e2e tests and
as for me should be run in separate step against dockerized env or
in case we want to check rpc handler we might want to rework it in order
to have interface for proper unit tests.
As for me this patchset at least makes as safe with not removing totally
previous tests and at the same time CircleCI will be happy now.
2019-09-18 18:21:16 +03:00
Vsevolod Brekelov
55dfc0bbbc storage: add seek implementation
during testing found missing Seek() implementation for inmemorydb
2019-09-18 18:20:41 +03:00
Roman Khimov
c68a254eba
Merge pull request #404 from nspcc-dev/move-user-docs
Move user docs, fix #339.
2019-09-18 12:50:06 +03:00
Roman Khimov
cb4be2ae4a storage: drop useless README, refs. #339 2019-09-17 17:30:05 +03:00
Vsevolod Brekelov
adc880d323 blockchain: server runs goroutine instead of blockchain init
rework initBlockChain in order to have controllable way of running
blockchain;
remove context from initBlockChain func;
2019-09-17 15:27:40 +03:00
Roman Khimov
d1a4e43c48 io: redo Serializable to return errors in BinReader/BinWriter
Further simplifies error handling.
2019-09-17 13:21:52 +03:00
Roman Khimov
0bb8950f89 make TXer and Payload implement Serializable
Both are duplicating Serializable at the moment, but let's keep them for the
future.
2019-09-17 13:21:52 +03:00
Roman Khimov
e299a44983 io: drop Size() method from Serializable and associated
It's no longer needed after the io.GetVarSize() improvement. It's duplicating
a lot of EncodeBinary() logic also.
2019-09-17 13:21:45 +03:00
Roman Khimov
0da9fe6946 io: move size calculator there
It's mostly used for Serializable and in other cases where one needs to
estimate binary-encoded size of the stucture. This also simplifies future
removal of the Size() from Serializable.
2019-09-16 23:39:54 +03:00
Roman Khimov
5bf00db2c9 io: move BinReader/BinWriter there, redo Serializable with it
The logic here is that we'll have all binary encoding/decoding done via our io
package, which simplifies error handling. This functionality doesn't belong to
util, so it's moved.

This also expands BufBinWriter with Reset() method to fit the needs of core
package.
2019-09-16 23:39:51 +03:00
Roman Khimov
f01354b5bd core: make TestHeaderEncodeDecode use assert for field comparisons 2019-09-16 23:38:48 +03:00
Roman Khimov
aacf58c9ab util: add 'constructors' for BinReader/BinWriter
And an additional BufBinWriter to ease buffer management.
2019-09-16 23:38:48 +03:00
Roman Khimov
c98a626871
Merge pull request #399 from nspcc-dev/refactor_store
storage: refactor store, add Close()
2019-09-16 18:59:34 +03:00
Vsevolod Brekelov
264dfef370 storage: close function
add close function to storage interface
add common defer function call which will close db connection
remove context as soon as it's not needed anymore
updated unit tests
2019-09-16 18:52:47 +03:00
Roman Khimov
adba9e11ee
Merge pull request #396 from nspcc-dev/network-reconnections-and-fixes
This one fixes #390 and some connected problems. After this patchset the node reconnects to some other nodes if anything goes wrong and it better senses when something goes wrong. It also fixes some block handling problems based on the testnet connection experience.
2019-09-16 16:57:10 +03:00
Roman Khimov
b263c97bbb core: add some mechanism to drop bad blocks from cache
If we can't persist anything from our cache there is something wrong with
it. Try to remediate some possible problems.
2019-09-16 16:26:30 +03:00
Vsevolod Brekelov
2cabd5b274 storage: add boltDB unit tests 2019-09-14 10:29:08 +03:00
Vsevolod Brekelov
bfe5fd538e storage: add boltDB support
closes #335
2019-09-14 10:28:43 +03:00
Roman Khimov
521a707e33 core: don't print contracts in persistBlock()
Yeah, this really needs more love, but for now just don't spit 'em out on the
screen, please.
2019-09-12 16:12:12 +03:00
Roman Khimov
4395cea344 storage: fix reading overlapping hash blocks from the DB
In the unlikely event of overlapping hash block written to the DB we might end
up with wrong hash list. That happened to me for some reason when synching
with the testnet leading to the following keys with respective values:
  150000 -> 2000 hashes
  152000 -> 2000 hashes
  153999 -> 2000 hashes

Reading it hashes number 153999 and 154000 got the same values and the chain
couldn't sync correctly.
2019-09-11 20:33:41 +03:00
Roman Khimov
4f60fd3e8e core: optimize persist() for the case when no next block found
If there is no next block in the cache, looping through the whole cache won't
help.
2019-09-11 20:32:30 +03:00
Roman Khimov
7812fb28ea core: optimize blockchain init with existing DB
There is no need to generate genesis block (it's already in the DB) and we can
add all hashes at once without iterating over them.
2019-09-11 20:28:49 +03:00
Vsevolod Brekelov
0f31e7bf7f unitTests: fix for failing test
should fix circleCI and now these tests using inmemory db
2019-09-10 23:21:26 +03:00
Vsevolod Brekelov
06ecb1855e storage: redis db unit test 2019-09-10 17:22:45 +03:00
Vsevolod Brekelov
4f680703a4 storage: multiple DB support and Redis support
add config which closes #336
add redis db support
2019-09-10 17:22:21 +03:00
Roman Khimov
b8b26d8e79 core: fix 'implementation' misspell in test code
Found in the Go report card.
2019-09-09 14:53:50 +03:00
Roman Khimov
fabd11699a *: gofmt -s
Appy gofmt for all the source tree.
2019-09-09 12:02:24 +03:00
Roman Khimov
2c3e92923f keys: simplify error handling for PublicKey() and associated
PublicKey() for PrivateKey now just can't fail and it makes no sense to return
an error from it. There is a lot of associated functionality for which this
also is true, so adjust it accordingly and simplify a lot of code.
2019-09-05 12:34:12 +03:00
Roman Khimov
0b884b92b3 crypto: use PrivateKey to generate a key pair
It makes no sense to provide an API for throw-away public keys, so obtain it
via a new real keypair generation where appropriate (and that's only needed
for testing).
2019-09-05 09:38:13 +03:00
Roman Khimov
22bea5eccf core: remove ineffective assignment from test code
Spotted by goreport:
  Line 89: warning: ineffectual assignment to hash (ineffassign)
2019-09-03 18:17:44 +03:00
Roman Khimov
db9e37b3bb *: fix golint's omit type suggestions
pkg/core/transaction/attribute.go:67:14: should omit type uint8 from declaration of var urllen; it will be inferred from the right-hand side
pkg/crypto/keys/publickey.go:184:8: should omit type []byte from declaration of var b; it will be inferred from the right-hand side
pkg/network/payload/version_test.go:15:12: should omit type bool from declaration of var relay; it will be inferred from the right-hand side

Refs. #213.
2019-09-03 18:16:48 +03:00
Roman Khimov
89ed00ddfb core: simplify GetTransationResults() based on golint suggestion
Golint:
  pkg/core/blockchain.go:796:9: if block ends with a return statement, so drop
    this else and outdent its block (move short variable declaration to its own
    line if necessary)

Refs. #213.
2019-09-03 18:16:48 +03:00
Roman Khimov
0c160a81fe transaction: use increments instead of '+1's
Fixes golint warning:
  should replace sz += 1 with sz++

Refs. #213.
2019-09-03 18:01:37 +03:00
Roman Khimov
a9b9c9226d *: add/fix godoc comments to satisfy golint
Fixes things like:
 * exported type/method/function X should have comment or be unexported
 * comment on exported type/method/function X should be of the form "X ..."
   (with optional leading article)

Refs. #213.
2019-09-03 17:57:51 +03:00
Roman Khimov
6767bb5446 transaction: move state test from dev 2019-08-30 19:35:57 +03:00
Roman Khimov
b8adc36e6a transaction: add missing alloc in state decoder
Fixes panic two lines below:
panic: runtime error: index out of range [recovered]
        panic: runtime error: index out of range
2019-08-30 19:34:26 +03:00
Roman Khimov
203aab6a7e transaction: add binary encoding for publish
And expand the test to check for it. Fixes #319.
2019-08-30 19:19:43 +03:00
Roman Khimov
af212f496e transaction: implement encoding for state 2019-08-30 19:13:19 +03:00
Roman Khimov
868c9d6e8c transaction: add miner tx test from dev 2019-08-30 18:53:26 +03:00
Roman Khimov
88a47aeeb4 transaction: move enrollment test from dev into the master 2019-08-30 18:28:12 +03:00
Roman Khimov
aa94ae70b6 transaction: move and adapt test from dev
Slightly increases coverage for transaction package.
2019-08-30 18:23:53 +03:00
Roman Khimov
d6686350ee transaction: auto-generate stringer interface for AttrUsage 2019-08-30 17:44:36 +03:00
Roman Khimov
3495fecc65 transaction: fix DescriptionURL attribute handling
Both dev and master actually did it wrong.
2019-08-30 16:51:18 +03:00
Roman Khimov
17679c3b94 core: add block test from dev branch
And drop it from _pkg.dev. Adapted for master.
2019-08-30 11:41:10 +03:00
Roman Khimov
683424cce8 transaction: implement proper Size() everywhere
Will be needed for the block test from `dev`.
2019-08-30 11:41:10 +03:00
Roman Khimov
200cce9f02 transaction: fix Size() calculation for attribute
It wasn't accounting for attr.Data length properly.
2019-08-30 11:41:10 +03:00
Roman Khimov
8ececdc9a7 transaction: fix invocation decoding for different versions
Quick fix similar to abc5833853 + uncomment the
test case for it.

Closes #173.
2019-08-30 11:41:10 +03:00
Roman Khimov
2ef2c0b84c transaction: properly print number in attribute log msg 2019-08-30 11:41:10 +03:00
Roman Khimov
c093f070d3 core: add EncodeBinary() support for the block 2019-08-30 11:41:10 +03:00
Vsevolod Brekelov
abc5833853 transaction: add version check for publish
-Version check done before reading NeedStorage flag
-Add testify according to review
2019-08-29 19:14:22 +03:00
Roman Khimov
361724a33e pkg/core: use util.binaryReader/Writer
Simplify error handling.
2019-08-28 19:45:17 +03:00
Roman Khimov
07096a551b util: move strange Read2000Uint256Hashes() into storage
It's the only user of it.
2019-08-28 14:38:57 +03:00
Roman Khimov
b77e533d13 crypto/wallet: move public/private key into the new keys package
And drop associated _pkg.dev remnants (refs. #307).

Original `dev` branch had two separate packages for public and private keys,
but those are so intertwined (`TestHelper` subpackage is a proof) that it's
better unite them and all associated code (like WIF and NEP-2) in one
package. This patch also:
 * creates internal `keytestcases` package to share things with wallet (maybe
   it'll be changed in some future)
 * ports some tests from `dev`
 * ports Verify() method for public key from `dev`
 * expands TestPrivateKey() with public key check
2019-08-27 17:45:51 +03:00
Evgeniy Kulikov
33f59debf2
store: Fix DataRace in MemoryStore
- Add RWMutex to MemoryStore struct
- Use Lock/Unlock for writing
- Use RLock/RUnlock for reading
- Fix #313 issue
2019-08-27 16:48:30 +03:00
Roman Khimov
ec7e17ffa6 pkg: make use of the new crypto/hash package
Simplifies a lot of code and removes some duplication. Unfortunately I had to
move test_util random functions in same commit to avoid cycle
dependencies. One of these random functions was also used in core/transaction
testing, to simplify things I've just dropped it there and used a static
string (which is nice to have for a test anyway).

There is still sha256 left in wallet (but it needs to pass Hash structure into
the signing function).
2019-08-26 13:32:19 +03:00
Roman Khimov
0cde8d962d fixed8: switch to more uniform function naming 2019-08-26 13:32:19 +03:00
Roman Khimov
d5d570f793 uint256: add Reverse(), change String() to be BE
This one makes a little more obvious that we're operating with LE
representations mostly. Refs. #307. See #314 also.
2019-08-26 13:32:19 +03:00
Roman Khimov
33c512032f vm: switch from Opcode to Instruction
Which matches the changes done in neo-storm and also make this codebase closer
to the `dev` branch (that also uses Instruction).
2019-08-14 15:40:31 +03:00
dauTT
095653af23 Implement rpc server method: sendrawtransaction (#174)
* Added new config attributes: 'SecondsPerBlock','LowPriorityThreshold'

* Added new files:

* Added new method: CompareTo

* Fixed empty Slice case

* Added new methods: LessThan, GreaterThan, Equal, CompareTo

* Added new method: InputIntersection

* Added MaxTransactionSize, GroupOutputByAssetID

* Added ned method: ScriptHash

* Added new method: IsDoubleSpend

* Refactor blockchainer, Added Feer interface, Verify and GetMemPool method

* 1) Added MemPool
2) Added new methods to satisfy the blockchainer interface: IsLowPriority, Verify, GetMemPool

* Added new methods: RelayTxn, RelayDirectly

* Fixed tests

* Implemented RPC server method sendrawtransaction

* Refactor getrawtransaction, sendrawtransaction in separate methods

* Moved 'secondsPerBlock' to config file

* Implemented Kim suggestions:
1) Fixed data race issues
2) refactor Verify method
3) Get rid of unused InputIntersection  method due to refactoring Verify method
4) Fixed bug in https://github.com/CityOfZion/neo-go/pull/174#discussion_r264108135
5) minor simplications of the code

* Fixed minor issues related to

1) space
2) getter methods do not need pointer on the receiver
3) error message
4) refactoring  CompareTo method in uint256.go

* Fixed small issues

* Use sync.RWMutex instead of sync.Mutex

* Refined (R)Lock/(R)Unlock

* return error instead of bool in Verify methods
2019-03-20 12:30:05 +00:00
dauTT
19201dcf52 Implemented rpc server method GetRawTransaction (#135)
* Added utility function GetVarSize

* 1) Added Size method: this implied that Fixed8 implements now the serializable interface. 2) Added few arithmetic operation (Add, Sub, div): this will be used to calculated networkfeeand feePerByte. Changed return value of the Value() method to int instead of int64. Modified fixed8_test accordingly.

* Implemented Size or MarshalJSON method.
- Structs accepting the Size method implement the serializable interface.
- Structs accepting the MarshalJSON method implements the customized json marshaller interface.

* Added fee calculation

* Implemented rcp server method GetRawTransaction

* Updated Tests

* Fixed:
1) NewFixed8 will accept as input int64
2) race condition affecting configDeafault, blockchainDefault

* Simplified Size calculation

* 1) Removed global variable blockchainDefault, configDefault
2) Extended Blockchainer interface to include the methods: References, FeePerByte, SystemFee, NetworkFee
3) Deleted fee_test.go, fee.go. Moved corresponding methods to blockchain_test.go and blockchain.go respectively
4) Amended tx_raw_output.go

* Simplified GetVarSize Method

* Replaced ValueAtAndType with ValueWithType

* Cosmetic changes + Added test case getrawtransaction_7

* Clean up Print statement

* Filled up keys

* Aligned verbose logic with the C#-neo implementation

* Implemented @Kim requests
Refactor server_test.go

* Small fixes

* Fixed verbose logic
Added more tests
Cosmetic changes

* Replaced assert.NoError with require.NoError

* Fixed tests by adding context.Background() as argument

* Fixed tests
2019-02-20 18:39:32 +01:00
Evgeniy Kulikov
cdba88b9f2 Code refactoring (#143)
- simplify code
- prealoc slices
- check errors in tests
- regexp Compile replaced with MustCompile
- uint* cannot be negative
2019-02-19 14:22:33 +01:00
Evgeniy Kulikov
9c24bf9139 Blockchain graceful shutdown (#139)
* Blockchain graceful shutdown

- fix #138
- stop blockchain goroutine
- close leveldb database

* fix possible context leak (go vet)
2019-02-19 12:48:48 +01:00
Evgeniy Kulikov
67cbcac643 Fix typos (#133)
* Fix typos

* revert chains/unit_testnet

* revert chains

* fix review comments (thx @AlexVanin)
2019-02-13 18:01:10 +00:00
Evgeniy Kulikov
630919bf7d Fix typos and warnings for GoReport / GolangCiLinter (#132)
- typos
- gofmt -s
- govet warnings
- golangci-lint run
2019-02-09 16:53:58 +01:00
dauTT
7e43717657 Implemented rcp method GetAccountState (#124)
* Implemented rcp method GetAccountState

* code clean up

* Removed empty line

* Used consistently github.com/pkg/errors package. Amended error message

* Get rid of fmt.Sprintf and use either errors.Errorf or errors.Wrapf

* cosmetic changes
2019-02-08 09:04:38 +01:00
Evgeniy Kulikov
f000b76879 [FIX] Formatting and code-style (#118)
* [FIX] Formatting and code-style

- gofmt
- import resort
- prealloc slices
- simplify code

* fix vet
2019-01-25 12:20:35 +01:00
dauTT
1360e1de68 Added rpc unit tests (#107)
* Fixed small incosistencies related to comments and variable name

* For testing purposes we have to move the method newBlockchain from the cli/server/server.go to  pkg/core/blockchain.go. In addition we have to rename it to NewBlockchainLevelDB in order to be able to export it and avoid naming conflicts. In future we still need to think how to switch between different blockchain implementation easily but for the time being this is not possible.

* Added unit tests for the rpc server

* Added unit_testnet chain fixture

* fixed port number

* Added errors handling

* move unit_testnet chain from 'cli/chains' to 'pkg/rpc/chains'
2019-01-22 13:14:52 +01:00
Anastasia Prasolova
74f0019df2 SendToAddress RPC call (#114)
* func to get private key from raw bytes

* added function to create raw transfer tx

* fixes

* more fixes

* prettify code and comments; neoscan interaction put in dedicated files
2018-12-21 10:32:18 +01:00
dauTT
e2f42e92a0 Implemented rcp method GetAssetState (#103)
* Fix missing dot in configPath

* Add rpc GetAssetState method

* Update rpc README.md

* Update version to 0.45.10
2018-11-26 22:12:33 +01:00
Evgeniy Kulikov
6ccb518ab0 Optimizations + some improvements (#105)
* Optimizations + some improvements

- optimized pkg/core/storage.HeaderHashes
- optimized pkg/rpc.performRequest (used json.Encoder)
- fixes for pkg/util.ReadVarUint and pkg/util.WriteVarUint
- optimized and fix fixed8 (Fixed8DecodeString / MarshalJSON) + tests
- optimized and fix uint160 (Bytes / Uint160DecodeString / Equal / MarshalJSON) + tests
- optimized and fix uint256 (Bytes / Equal / MarshalJSON) + tests
- preallocate for pkg/vm.buildStackOutput
- add go.mod / go.sum

* update version
2018-11-26 16:56:45 +01:00
Anthony De Meulemeester
2cdfee211a
Persisting more states (#71)
* added persistence of assets and spentcoins.

* contract params

* bumped version
2018-04-16 22:15:30 +02:00
Anthony De Meulemeester
ab2568cc51
Fixed some networking issues (#68)
* Faster persist timer

* fixed networking issues.
2018-04-13 12:14:08 +02:00
Anthony De Meulemeester
b2021c126e
Tweaks for network and storage (#66)
* Made Encode/Decode message public.

* Added Redis storage driver and made some optimizations for the initialising the blockchain

* removed log lines in tcp_peer

* Added missing comments on exported methods.

* bumped version
2018-04-09 18:58:09 +02:00
Anthony De Meulemeester
931388b687
Cross platform virtual machine implementation (#60)
* Virtual machine for the NEO blockhain.

* fixed big.Int numeric operation pointer issue.

* added appcall

* Added README for vm package.

* removed main.go

* started VM cli (prompt) integration

* added support for printing the stack.

* moved cli to vm package

* fixed vet errors

* updated readme

* added more test for VM and fixed some edge cases.

* bumped version -> 0.37.0
2018-03-30 18:15:06 +02:00
Steven Jack
0b023c5c5c Small RPC improvements (#57)
* Few tweaks to improve output of `getblock`

* Adds few more fields and corrects witness

* Bumps version

* Only reverse when marshalling for moment

* Adds README for rpc package

* Few updates

* Typo

* Adds link in main readme

* Fix readme link
2018-03-30 08:15:03 +02:00
Anthony De Meulemeester
34bd9d31ac
Compiler arrays (#49)
* implemented operation and param flags in the cli invoke cmd.

* reverted prev changes and added debug flag for compiling.

* change transactionType variable to Type, for package convention

* index support for arrays.

* implemented builtin (len) for the compiler.

* bumped version -> 0.35.0

* updated compiler README and changed invoke to testinvoke.
2018-03-25 18:21:00 +02:00
Anthony De Meulemeester
94672cb9cc
Persistance (#53)
* added publish TX for backwards compat.

* lowered the prototick for faster block syncing

* print useragent on startup

* added createMultiRedeemScript for genesis block generation.

* building genesis block from scratch.

* implemented merkle tree.

* starting blockhain with generated genesis hash

* Fixed bug in unspent coin state.

* fixed broken tests after genesis block.

* removed log line.

* bumped version -> 0.34.0
2018-03-25 12:45:54 +02:00
Steven Jack
19a430b262 RCP server (#50)
* Adds basic RPC supporting files

* Adds interrupt handling and error chan

* Add getblock RPC method

* Update request structure

* Update names of nodes

* Allow bad addresses to be registered in discovery externally

* Small tidy up

* Few tweaks

* Check if error is close error in tcp transport

* Fix tests

* Fix priv port

* Small tweak to param name

* Comment fix

* Remove version from server

* Moves submitblock to TODO block

* Remove old field

* Bumps version and fix hex issues
2018-03-23 21:36:59 +01:00
Anthony De Meulemeester
52fa41a12a
Persist transactions (#51)
* added account_state + changed ECPoint to PublicKey

* account state persist

* in depth test for existing accounts.

* implemented GetTransaction.

* added enrollment TX

* added persist of accounts and unspent coins

* bumped version -> 0.32.0
2018-03-21 17:11:04 +01:00
Anthony De Meulemeester
a67728628e
Persist blockchain with leveldb on disk (#48)
* Created test_data folder with block json files for testing + create separate file for block base.

* Fixed bug in WriteVarUint + Trim logic + unit tests

* Refactored store and add more tests for it.

* restore headerList from chain file

* Fix tx decode bug + lots of housekeeping.

* Implemented Node restore state from chain file.

* Created standalone package for storage. Added couple more methods to Batch and Store interfaces.

* Block persisting + tests

* bumped version -> 0.31.0
2018-03-17 12:53:21 +01:00
Anthony De Meulemeester
aa4bc1b6e8
Node improvements (#47)
* block partial persist

* replaced refactored files with old one.

* removed gokit/log from deps

* Tweaks to not overburden remote nodes with getheaders/getblocks

* Changed Transporter interface to not take the server as argument due to a cause of race warning from the compiler

* started server test suite

* more test + return errors from message handlers

* removed --race from build

* Little improvements.
2018-03-14 10:36:59 +01:00
Anthony De Meulemeester
aa4bd34b6b
Node network improvements (#45)
* small improvements.

* Fixed datarace + cleanup node and peer

* bumped version.

* removed race flag to pass build
2018-03-10 13:04:06 +01:00
Anthony De Meulemeester
4023661cf1
Refactor of the Go node (#44)
* added headersOp for safely processing headers

* Better handling of protocol messages.

* housekeeping + cleanup tests

* Added more blockchain logic + unit tests

* fixed unreachable error.

* added structured logging for all (node) components.

* added relay flag + bumped version
2018-03-09 16:55:25 +01:00
Anthony De Meulemeester
1a1a19da7d
Smartcontract (#39)
* deleted transfer_output added asset type and transaction result to core

* removed writing 0x00 when buffer length is 0

* Refactored emit into VM package + moved tx to own package.

* implemented transaction along with claimTransaction.

* refactored naming of transaction + added decode address for uint160 types

* removed unnecessary folder and files.

* transaction/smartcontract logic

* bumped version 0.24.0
2018-03-04 14:56:49 +01:00
Anthony De Meulemeester
f3f6662fc9
Base wallet implementation (#35)
* Initial draft of the neo-go wallet

* Cleanup + more test for util package

* integrated wallet into neo-cli partially

* base wallet implementation + smartcontract code.
2018-03-02 16:24:09 +01:00
Anthony De Meulemeester
b6d8271b8d
Fixed header sync issue (#17)
* headers can now sync till infinity

* fixed empty hashStop getBlock payload + test

* added more test + more binary decoding/encoding

* bump version
2018-02-07 15:16:50 +01:00
Anthony De Meulemeester
046494dd68
Implemented processing headers + added leveldb as a dependency. (#16)
* Implemented processing headers + added leveldb as a dependency.

* version 0.7.0

* put glide get and install in build_cli section
2018-02-06 07:43:32 +01:00
Anthony De Meulemeester
628656483a
bug fixes (TCP + uint256) and started core part (#14)
* Fixed TCP read + Uint256 reversed array + started on some core pieces

* Disabled some debug output + muted test

* 0.5.0
2018-02-04 20:54:51 +01:00
Anthony De Meulemeester
66c8fc8012
merge original into coz repo (#10)
merged with the original repo.
2018-02-01 21:28:45 +01:00
Charlie Revett
dd94086a22
CircleCI 2 & Releases (#9) 2018-02-01 10:54:23 -08:00
Charlie Revett
a95ce31176
Separate TransactionType to new file (#8) 2018-02-01 10:05:56 -08:00
anthdm
55b4ab4192 Block binary decoding + transaction types 2018-01-31 11:47:54 +01:00
anthdm
0c9d2dd04e Block message + handle the length of the user agent better. 2018-01-31 09:27:08 +01:00
anthdm
e9f9354b86 added core block type 2018-01-30 11:56:36 +01:00
anthdm
536a499236 initial commit. 2018-01-26 19:04:13 +01:00