Commit graph

206 commits

Author SHA1 Message Date
Vsevolod Brekelov
063fe5be9d unspentcoinstate: fix branch never executed 2019-10-21 12:40:11 +03:00
Roman Khimov
b533dfceba core: don't panic on init when there are less than 2000 hashes
If you're to sync less than 2000 headers no batched header key-value is
gonna be written into the DB and init() would panic because
bc.headerList.Len() would return 0. Use genesis block as a target in this
case.
2019-10-21 08:37:01 +03:00
Vsevolod Brekelov
fe39c565b9 codestyle: redundant type conversion 2019-10-18 18:39:31 +03:00
Vsevolod Brekelov
1afdb895a1 codestyle: declare empty slice
https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices
2019-10-18 18:38:33 +03:00
Vsevolod Brekelov
ced2022273 storage: change println error for botldb to log 2019-10-17 18:23:49 +03:00
Roman Khimov
3fc2bf5452 *: fix some misspellings
Goreport:
   neo-go/pkg/core/contract_state_test.go
        Line 21: warning: "Contracto" is a misspelling of "Contraction" (misspell)
        Line 64: warning: "Contracto" is a misspelling of "Contraction" (misspell)

   neo-go/pkg/core/interop_neo.go
        Line 420: warning: "succeedes" is a misspelling of "succeeds" (misspell)

   neo-go/pkg/network/discovery.go
        Line 118: warning: "succeded" is a misspelling of "succeeded" (misspell)
        Line 128: warning: "successfuly" is a misspelling of "successfully" (misspell)
2019-10-17 12:30:24 +03:00
Roman Khimov
1b5ea67737 core: fix ineffassign suggestion
Goreport:
  Line 467: warning: ineffectual assignment to persisted (ineffassign)
2019-10-17 12:27:15 +03:00
Roman Khimov
288000a8af storage: drop useless nil assignment in leveldb code
golint suggests:
   Line 24: warning: should drop = nil from declaration of var opts; it is the zero value (golint)
2019-10-17 12:18:16 +03:00
Roman Khimov
2245fedbb1 core: deduplicate state commit methods, use interim MemCachedStore
commit methods duplicated putSmthIntoStore functions, but have MemCachedStore
now that can easily substitute for a Batch, especially given that interop
needs something like that for its storage purposes anyway.
2019-10-16 17:33:45 +03:00
Roman Khimov
fc0031e5aa core: move write caching layer into MemCacheStore
Simplify Blockchain and associated functions, deduplicate code, fix Get() and
Seek() implementations.
2019-10-16 17:33:45 +03:00
Roman Khimov
8926cbe368 core: fix potential data race in logging code
Spotted in CircleCI build with Go 1.12:

WARNING: DATA RACE
Write at 0x00c00011095c by goroutine 88:
  sync/atomic.SwapInt32()
      /usr/local/go/src/runtime/race_amd64.s:249 +0xb
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).persist()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:483 +0x172
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run.func2()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:191 +0x50

Previous read at 0x00c00011095c by goroutine 64:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).persist()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:492 +0x4be
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run.func2()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:191 +0x50

Goroutine 88 (running) created at:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:190 +0x264

Goroutine 64 (finished) created at:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:190 +0x264
2019-10-15 18:58:22 +03:00
Roman Khimov
a6610ba082 core: verify blocks, fix #12
This adds the following verifications:
 * merkleroot check
 * index check
 * timestamp check
 * witnesses verification

VerifyWitnesses is also renamed to verifyTxWitnesses here to not confuse it
with verifyBlockWitnesse and to hide it from external access (no users at the
moment).
2019-10-15 18:58:17 +03:00
Roman Khimov
03c20f1876 core: make ContractState receiver names consistent
Linter isn't happy with our recent changes:

pkg/core/contract_state.go:109:1: receiver name cs should be consistent with previous receiver name a for ContractState
pkg/core/contract_state.go:114:1: receiver name cs should be consistent with previous receiver name a for ContractState
pkg/core/contract_state.go:119:1: receiver name cs should be consistent with previous receiver name a for ContractState

But actually `a` here most probably is a copy-paste from AssetState methods,
so fit the old code to match the new one.
2019-10-15 12:56:25 +03:00
Roman Khimov
aec6a5f029 core/config: make block and transaction verification configurable
Enable transaction verification for privnets and tests, testnet can't
successfuly verify block number 316711 with it enabled and mainnet stops at
105829.
2019-10-15 12:56:25 +03:00
Roman Khimov
56dcff2894 core: invoke contracts for Invocation TXes
They can have some side-effects that future invocations rely on.
2019-10-15 12:56:25 +03:00
Roman Khimov
74590551c4 core: add some interops
This also changes Verify to VerifyTx and VerifyWitnesses, because there is a
need to pass a block for some interop functions.
2019-10-15 12:56:25 +03:00
Roman Khimov
667f346c04 core: improve error message in AddBlock test 2019-10-15 12:56:25 +03:00
Roman Khimov
8266a5ce19 core: export GetContractState/GetScriptHashesForVerifying via Blockchainer
These are gonna be used by interops and are also useful in general.
2019-10-15 12:56:25 +03:00
Roman Khimov
19fd7f844e core: add GetUnspentCoinState() for future interops 2019-10-15 12:56:25 +03:00
Roman Khimov
bfddf9b3f6 core: implement StorageItem for future interops 2019-10-15 12:56:25 +03:00
Roman Khimov
2156d5db73 core: add put/delete functions for ContractState
These are gonna be used by interops.
2019-10-15 12:56:25 +03:00
Roman Khimov
58d9b79fe3 core: avoid duplicates in GetScriptHashesForVerifyingClaim()
As they can break VerifyWitnesses().
2019-10-15 12:56:25 +03:00
Roman Khimov
35824728eb core: add putAssetStateIntoStore()
Will be used by interops.
2019-10-15 12:56:25 +03:00
Roman Khimov
96934cfeab core: return error from verifyOutputs()
Make debugging easier.
2019-10-15 12:56:25 +03:00
Roman Khimov
ab4ff79e02 core: fix typo in GetTransactionResults() 2019-10-15 12:56:25 +03:00
Roman Khimov
cffe8d0ee2 core: return error from verifyResults()
Make it easier to find out what the real problem is.
2019-10-15 12:56:25 +03:00
Roman Khimov
16bc5296cb block: return error from Verify
Don't hide real problem behind the bool value. Makes it easier to identify
problems when looking at log messages.
2019-10-15 12:56:25 +03:00
Roman Khimov
258f397b9a core: append transactions to the block in GetBlock()
We want to get a full block, so it has to have transactions
inside. Unfortunately our tests were used to this wrong behavior and utilized
completely bogus transactions without data that couldn't be persisted, so fix
that also.
2019-10-15 12:56:25 +03:00
Roman Khimov
238c590ddb core: fix contract state's Properties to use PropertyState
PublishTX only had one of these flags, but newer contracts (created via the
interop function) can have more and these flags are aggregated into one field
that uses PropertyState enumeration (it's used to publish contract, so
supposedly it's also a nice choice for contract state storage).
2019-10-15 12:56:25 +03:00
Roman Khimov
78861485b6 storage: simplify MemoryBatch
It's used a lot and it looks a lot like MemoryStore, it just needs not to
return errors from Put and Delete, so make it use MemoryStore internally with
adjusted interface.
2019-10-15 12:56:25 +03:00
Roman Khimov
13bf2618ef storage: improve PutBatch for MemoryStore
Make it look more like a real transaction, put/delete things with a single
lock. Make a copy of value in Put also, just for safety purposes, no one knows
how this value slice can be used after the Put.
2019-10-15 12:56:25 +03:00
Roman Khimov
3ada92944a storage: drop Len from the Batch interface
It's almost meaningless now and we can easily live without it.
2019-10-15 12:56:25 +03:00
Roman Khimov
e111892653 storage: redo DB testing
Make generic tests for all Store implementations, deduplicate tests. Implement
Delete() tests, test Seek() better, add LevelDB tests (finally!).
2019-10-15 12:56:25 +03:00
Roman Khimov
48b6a427cf storage: add Delete method for Batch and Store
It's gonna be used by Storage and Contract interops, both can delete their
data.
2019-10-07 19:40:11 +03:00
Roman Khimov
add9368e9d storage: use strings as keys for memory batch
Using pointers is just plain wrong here, because the batch can be updated with
newer values for the same keys.

Fixes Seek() to use HasPrefix also because this is the intended behavior.
2019-10-07 17:05:53 +03:00
Roman Khimov
1bf232ad50 vm: introduce TryBool() for Element and use it in VerifyWitnesses
Script can return non-bool results that can still be converted to bool
according to the usual VM rules. Unfortunately Bool() panics if this
conversion fails which is OK for things done in vm.execute(), but certainly
not for VerifyWitnesses(), thus there is a need for TryBool() that will just
return an error in this case.
2019-10-04 16:13:39 +03:00
Roman Khimov
ceca9cdb67 core/vm: implement contract storage and script retrieval
Fixes script invocations via the APPCALL instruction. Adjust contract state
field types accordingly.
2019-10-04 16:13:39 +03:00
Roman Khimov
b7c1efe7e3 core: fix References() result key type
If the block references two ouputs in some other transaction the code failed
to verify it because of key collision. C# code implements it properly by using
full CoinReference type as a key, so let's do it in a similar fashion.
2019-10-01 13:41:26 +03:00
Roman Khimov
8537700b7b core: sort hashes and witnesses in VerifyWitnesses()
Fixes failure to verify some multi-witnesses transactions in Testnet chain. C#
code contains similar logic.
2019-10-01 13:41:26 +03:00
Roman Khimov
dd0dd2d031 core: add claim-specific GetScriptHashesForVerifying()
Claim transactions have different logic in C# node, so we need to
implement it too. It's not the most elegant way to fix it, but let's make it
work first and then refactor if and where needed. Fixes verification of Claim
transactions.
2019-10-01 13:41:26 +03:00
Roman Khimov
854fb187a3 core: verify transactions in AddBlock() 2019-10-01 13:41:26 +03:00
Roman Khimov
1095abb04c core: fix default asset expiration to match C# code
Fixes bogus verification failures for non-native assets.
2019-10-01 13:41:26 +03:00
Roman Khimov
832c56f97d core: fix DutyFlag check in GetScriptHashesForVerifying()
It's a flag (used by Share and Invoice asset types), so it should be checked
like a flag, the same way C# node does.
2019-10-01 13:41:26 +03:00
Roman Khimov
4ae18e8ffc block: check for Transaction length before messing with txes
Fixes panic two lines below. Blocks without transactions are invalid by
definition, so there is a need to adjust tests accordingly.
2019-10-01 13:41:26 +03:00
Roman Khimov
7779ad6c28 storage: always return ErrKeyNotFound when key is not found
It was true for MemoryStore and BoltDB, but not for LevelDB and Redis.
2019-09-27 15:42:36 +03:00
Roman Khimov
5ff1399d54 storage: unexport MemoryStore mutex
It shouldn't be available from outside.
2019-09-27 15:42:36 +03:00
Roman Khimov
3c40a53c4d core: no need to Close() Blockchain in tests
It's a Store method actually and for every Blockchain that uses Run() the
Close() will be handled automatically on exit because of context magic.
2019-09-27 15:42:35 +03:00
Roman Khimov
0dbdbb9c2a storage: remove RedisBatch in favor of MemoryBatch
They're also almost the same.
2019-09-27 15:42:35 +03:00
Roman Khimov
af557cea19 storage: deduplicate BoltDBBatch/MemoryBatch
BoltDB doesn't have internal batching mechanism, thus we have a substitute for
it, but this substitute is absolutely identical to MemoryBatch, so it's better
to unify them and import ac5d2f94d3 fix into the
MemoryBatch.
2019-09-27 15:42:35 +03:00
Roman Khimov
8bf37f45fc storage: remove impossible error branch from MemoryStore
MemoryStore is always successful, so this branch makes no sense.
2019-09-27 15:42:35 +03:00
Roman Khimov
4a732d97ac storage: add comments about errors from MemoryStore
It never returns errors for some operations and this knowledge can be used by
other code.
2019-09-27 15:42:35 +03:00
Roman Khimov
920d7c610c core: remove blockCache, use MemoryStore, redesign persist()
Commit 578ac414d4 was wrong in that it saved
only a part of the block, so depending on how you use blockchain, you may
still see that the block was not really processed properly. To really fix it
this commit introduces intermediate storage layer in form of memStore, which
actually is a MemoryStore that supports full Store API (thus easily fitting
into the existing code) and one extension that allows it to flush its data to
some other Store.

It also changes AddBlock() semantics in that it only accepts now successive
blocks, but when it does it guarantees that they're properly added into the
Blockchain and can be referred to in any way. Pending block queing is now
moved into the server (see 8c0c055ac657813fe3ed10257bce199e9527d5ed).

So the only thing done with persist() now is just a move from memStore to
Store which probably should've always been the case (notice also that
previously headers and some other metadata was written into the Store
bypassing caching/batching mechanism thus leading to some inefficiency).
2019-09-27 15:42:35 +03:00
Roman Khimov
c531dc0bde network: add block queue
This one will replace blockCache in Blockchain itself as it can and should be
external from it. The idea is that we only feed successive blocks into the
Blockchain and it only stores valid proper Blockchain and nothing else.
2019-09-27 13:00:09 +03:00
Roman Khimov
903173c991 core: fix deadlock in headers list init
This probably is a problem since adc880d323,
because AddHeaders() uses headersOp channel and its reader is only ran in
Run().
2019-09-25 17:52:46 +03:00
Roman Khimov
578ac414d4 core: make unpersisted blocks/txs available in Blockchain
This changes the Blockchain to also return unpersisted (theoretically, verified
in the AddBlock!) blocks and transactions, making Add/Get interfaces
symmetrical. It allows to turn Persist into internal method again and makes it
possible to enable transaction check in GetBlock(), thus fixing #366.
2019-09-25 17:46:28 +03:00
Roman Khimov
8a457ef540 transaction: add a check for createHash() invocation
It should work in general.
2019-09-24 13:01:54 +03:00
Roman Khimov
679e01bd0f core: implement witness verification logic
Fix #368.
2019-09-24 13:01:54 +03:00
Roman Khimov
ac5d2f94d3 storage: fix BoltDB batched Put()
It must copy both the value and the key because they can be reused for other
purposes between Put() and PutBatch(). This actually happens with values in
headers processing, leading to wrong data being written into the DB.

Extend the batch test to check for that.
2019-09-23 09:27:18 +03:00
Roman Khimov
4b4bac675b core: don't print persisting errors twice
As they're already printed in the calling goroutine. And they're alse not
always useful when printed by Persist().
2019-09-22 20:08:15 +03:00
Roman Khimov
102c926ef3 core: don't print useless persist messages with nil errors
Errors should be printed only when there are errors.
2019-09-22 20:07:49 +03:00
Roman Khimov
5cddfe071b
Merge pull request #403 from nspcc-dev/fix_rpctest
Fix rpctest, fix #353, fix #305.
2019-09-18 23:14:37 +03:00
Vsevolod Brekelov
100fee164b unitTest: reworked RPC unit test
earlier we had an issue with failing test in #353 and other one #305.
Reworked these test to have in-memory database. This led to multiple
changes: made some functions like Hash and Persist public(otherwise
it's not possible to control state of the blockchain); removed
unit_tests storage package which was used mainly for leveldb in unit
tests.
I see these tests not really good since they look like e2e tests and
as for me should be run in separate step against dockerized env or
in case we want to check rpc handler we might want to rework it in order
to have interface for proper unit tests.
As for me this patchset at least makes as safe with not removing totally
previous tests and at the same time CircleCI will be happy now.
2019-09-18 18:21:16 +03:00
Vsevolod Brekelov
55dfc0bbbc storage: add seek implementation
during testing found missing Seek() implementation for inmemorydb
2019-09-18 18:20:41 +03:00
Roman Khimov
c68a254eba
Merge pull request #404 from nspcc-dev/move-user-docs
Move user docs, fix #339.
2019-09-18 12:50:06 +03:00
Roman Khimov
cb4be2ae4a storage: drop useless README, refs. #339 2019-09-17 17:30:05 +03:00
Vsevolod Brekelov
adc880d323 blockchain: server runs goroutine instead of blockchain init
rework initBlockChain in order to have controllable way of running
blockchain;
remove context from initBlockChain func;
2019-09-17 15:27:40 +03:00
Roman Khimov
d1a4e43c48 io: redo Serializable to return errors in BinReader/BinWriter
Further simplifies error handling.
2019-09-17 13:21:52 +03:00
Roman Khimov
0bb8950f89 make TXer and Payload implement Serializable
Both are duplicating Serializable at the moment, but let's keep them for the
future.
2019-09-17 13:21:52 +03:00
Roman Khimov
e299a44983 io: drop Size() method from Serializable and associated
It's no longer needed after the io.GetVarSize() improvement. It's duplicating
a lot of EncodeBinary() logic also.
2019-09-17 13:21:45 +03:00
Roman Khimov
0da9fe6946 io: move size calculator there
It's mostly used for Serializable and in other cases where one needs to
estimate binary-encoded size of the stucture. This also simplifies future
removal of the Size() from Serializable.
2019-09-16 23:39:54 +03:00
Roman Khimov
5bf00db2c9 io: move BinReader/BinWriter there, redo Serializable with it
The logic here is that we'll have all binary encoding/decoding done via our io
package, which simplifies error handling. This functionality doesn't belong to
util, so it's moved.

This also expands BufBinWriter with Reset() method to fit the needs of core
package.
2019-09-16 23:39:51 +03:00
Roman Khimov
f01354b5bd core: make TestHeaderEncodeDecode use assert for field comparisons 2019-09-16 23:38:48 +03:00
Roman Khimov
aacf58c9ab util: add 'constructors' for BinReader/BinWriter
And an additional BufBinWriter to ease buffer management.
2019-09-16 23:38:48 +03:00
Roman Khimov
c98a626871
Merge pull request #399 from nspcc-dev/refactor_store
storage: refactor store, add Close()
2019-09-16 18:59:34 +03:00
Vsevolod Brekelov
264dfef370 storage: close function
add close function to storage interface
add common defer function call which will close db connection
remove context as soon as it's not needed anymore
updated unit tests
2019-09-16 18:52:47 +03:00
Roman Khimov
adba9e11ee
Merge pull request #396 from nspcc-dev/network-reconnections-and-fixes
This one fixes #390 and some connected problems. After this patchset the node reconnects to some other nodes if anything goes wrong and it better senses when something goes wrong. It also fixes some block handling problems based on the testnet connection experience.
2019-09-16 16:57:10 +03:00
Roman Khimov
b263c97bbb core: add some mechanism to drop bad blocks from cache
If we can't persist anything from our cache there is something wrong with
it. Try to remediate some possible problems.
2019-09-16 16:26:30 +03:00
Vsevolod Brekelov
2cabd5b274 storage: add boltDB unit tests 2019-09-14 10:29:08 +03:00
Vsevolod Brekelov
bfe5fd538e storage: add boltDB support
closes #335
2019-09-14 10:28:43 +03:00
Roman Khimov
521a707e33 core: don't print contracts in persistBlock()
Yeah, this really needs more love, but for now just don't spit 'em out on the
screen, please.
2019-09-12 16:12:12 +03:00
Roman Khimov
4395cea344 storage: fix reading overlapping hash blocks from the DB
In the unlikely event of overlapping hash block written to the DB we might end
up with wrong hash list. That happened to me for some reason when synching
with the testnet leading to the following keys with respective values:
  150000 -> 2000 hashes
  152000 -> 2000 hashes
  153999 -> 2000 hashes

Reading it hashes number 153999 and 154000 got the same values and the chain
couldn't sync correctly.
2019-09-11 20:33:41 +03:00
Roman Khimov
4f60fd3e8e core: optimize persist() for the case when no next block found
If there is no next block in the cache, looping through the whole cache won't
help.
2019-09-11 20:32:30 +03:00
Roman Khimov
7812fb28ea core: optimize blockchain init with existing DB
There is no need to generate genesis block (it's already in the DB) and we can
add all hashes at once without iterating over them.
2019-09-11 20:28:49 +03:00
Vsevolod Brekelov
0f31e7bf7f unitTests: fix for failing test
should fix circleCI and now these tests using inmemory db
2019-09-10 23:21:26 +03:00
Vsevolod Brekelov
06ecb1855e storage: redis db unit test 2019-09-10 17:22:45 +03:00
Vsevolod Brekelov
4f680703a4 storage: multiple DB support and Redis support
add config which closes #336
add redis db support
2019-09-10 17:22:21 +03:00
Roman Khimov
b8b26d8e79 core: fix 'implementation' misspell in test code
Found in the Go report card.
2019-09-09 14:53:50 +03:00
Roman Khimov
fabd11699a *: gofmt -s
Appy gofmt for all the source tree.
2019-09-09 12:02:24 +03:00
Roman Khimov
2c3e92923f keys: simplify error handling for PublicKey() and associated
PublicKey() for PrivateKey now just can't fail and it makes no sense to return
an error from it. There is a lot of associated functionality for which this
also is true, so adjust it accordingly and simplify a lot of code.
2019-09-05 12:34:12 +03:00
Roman Khimov
0b884b92b3 crypto: use PrivateKey to generate a key pair
It makes no sense to provide an API for throw-away public keys, so obtain it
via a new real keypair generation where appropriate (and that's only needed
for testing).
2019-09-05 09:38:13 +03:00
Roman Khimov
22bea5eccf core: remove ineffective assignment from test code
Spotted by goreport:
  Line 89: warning: ineffectual assignment to hash (ineffassign)
2019-09-03 18:17:44 +03:00
Roman Khimov
db9e37b3bb *: fix golint's omit type suggestions
pkg/core/transaction/attribute.go:67:14: should omit type uint8 from declaration of var urllen; it will be inferred from the right-hand side
pkg/crypto/keys/publickey.go:184:8: should omit type []byte from declaration of var b; it will be inferred from the right-hand side
pkg/network/payload/version_test.go:15:12: should omit type bool from declaration of var relay; it will be inferred from the right-hand side

Refs. #213.
2019-09-03 18:16:48 +03:00
Roman Khimov
89ed00ddfb core: simplify GetTransationResults() based on golint suggestion
Golint:
  pkg/core/blockchain.go:796:9: if block ends with a return statement, so drop
    this else and outdent its block (move short variable declaration to its own
    line if necessary)

Refs. #213.
2019-09-03 18:16:48 +03:00
Roman Khimov
0c160a81fe transaction: use increments instead of '+1's
Fixes golint warning:
  should replace sz += 1 with sz++

Refs. #213.
2019-09-03 18:01:37 +03:00
Roman Khimov
a9b9c9226d *: add/fix godoc comments to satisfy golint
Fixes things like:
 * exported type/method/function X should have comment or be unexported
 * comment on exported type/method/function X should be of the form "X ..."
   (with optional leading article)

Refs. #213.
2019-09-03 17:57:51 +03:00
Roman Khimov
6767bb5446 transaction: move state test from dev 2019-08-30 19:35:57 +03:00
Roman Khimov
b8adc36e6a transaction: add missing alloc in state decoder
Fixes panic two lines below:
panic: runtime error: index out of range [recovered]
        panic: runtime error: index out of range
2019-08-30 19:34:26 +03:00
Roman Khimov
203aab6a7e transaction: add binary encoding for publish
And expand the test to check for it. Fixes #319.
2019-08-30 19:19:43 +03:00
Roman Khimov
af212f496e transaction: implement encoding for state 2019-08-30 19:13:19 +03:00
Roman Khimov
868c9d6e8c transaction: add miner tx test from dev 2019-08-30 18:53:26 +03:00