Commit graph

345 commits

Author SHA1 Message Date
Roman Khimov
54d888ba70 io: add type-specific read/write methods
This seriously improves the serialization/deserialization performance for
several reasons:
 * no time spent in `binary` reflection
 * no memory allocations being made on every read/write
 * uses fast ReadBytes everywhere it's appropriate

It also makes Fixed8 Serializable just for convenience.
2019-12-12 20:19:50 +03:00
Roman Khimov
89d7f6d26e core/tx: microoptimize block/tx hashing
Don't hash the data twice.
2019-12-12 18:01:30 +03:00
Vsevolod Brekelov
c93a8d2bc4 core: extracted same logic to separate methods 2019-12-11 13:14:45 +03:00
Vsevolod Brekelov
c1f39d5c7b internal: moved testutil method to internal package 2019-12-11 13:14:43 +03:00
Vsevolod Brekelov
2d42b14a1d core: renames entities-> state and removed State prefix 2019-12-11 13:14:18 +03:00
Vsevolod Brekelov
8809fe437d core: unit tests for dao 2019-12-11 13:13:51 +03:00
Vsevolod Brekelov
c0e59ebd4e core: unit tests for entities 2019-12-11 13:05:31 +03:00
Vsevolod Brekelov
c7ac4b6dff core: fix encoding and decoding for notification event 2019-12-11 13:05:31 +03:00
Vsevolod Brekelov
ec17654986 core: refactoring blockchain state and storage
add dao which takes care about all CRUD operations on storage
remove blockchain state since everything is stored on change
remove storage operations from structs(entities)
move structs to entities package
2019-12-11 13:05:31 +03:00
Roman Khimov
d0f9a28196 vm/core: improve block import speed with PublicKey caching
This change (closely related to the neo-project/neo#1321 proposal) speeds up
1.4M mainnet blocks import by 30%. Basically, we're eliminating key decoding
for block's multisignature that has the same keys most of the time.

Things I don't like about this patch:
 * yet another parameter for verifyHashAgainstScript()
 * vm keys are not copied in/out

But it's rather simple and solves the problem for this particular case, so I
think it's worth it.
2019-12-10 19:13:29 +03:00
Roman Khimov
f1856bfa8b core/tx: remove publickey indirection from assets and txes
It makes very little sense having pointers here, these structures MUST have
some kind of key and this key is not gonna be wandering somewhere on its
own. Fixes a part of #519.
2019-12-09 18:33:04 +03:00
Roman Khimov
5b6c5af704 *: implement EncodeBinary with pointer receivers where appropriate
Everywhere except ParamType (which is just a byte), reduce copying things
around for no real reason.
2019-12-09 18:25:15 +03:00
Roman Khimov
7e371588a7 core/tx: remove one layer of indirection for scripts and inouts
It reduces heap pressure a little for these elements as we don't have to
allocate/free them individually. And they're directly tied to transactions or
block, not being shared or anything like that, so it makes little sense for
them to be pointer-based. It only makes building transactions a little easier,
but that's obviously a minor usecase.
2019-12-09 17:14:10 +03:00
Evgenii Stratonikov
fccb008594 io: implement ReadBytes() 2019-12-09 15:00:15 +03:00
Evgenii Stratonikov
838050f8b5 io: rename ReadBytes() to ReadVarBytes() 2019-12-09 15:00:15 +03:00
Roman Khimov
e4d821f32d
Merge pull request #546 from nspcc-dev/write-optimizations
Write optimizations
2019-12-06 19:40:38 +03:00
Roman Khimov
9992a98007 core: optimize balance utxo removal
One of my samples had 8 (out of 30) seconds spent here, but values of 100ms
are more typical. After this change it becomes invisible.
2019-12-06 18:46:24 +03:00
Roman Khimov
844491d365 *: use more efficient WriteBytes where appropriate
Before this patch on block import we could easily be spending more than 6
seconds out of 30 in Uint256 encoding for UnspentBalance, now it's completely
off the radar.
2019-12-06 18:22:21 +03:00
Evgenii Stratonikov
7179e4ba9f util: add LE suffix to Uint256 methods 2019-12-06 12:16:55 +03:00
Evgenii Stratonikov
57efad912c util: add LE suffix to Uint160 methods 2019-12-06 12:16:55 +03:00
Roman Khimov
8beb135829
Merge pull request #542 from nspcc-dev/fix-storage-interop-reads
core: fix wrong data being read in interops (part of #501)
2019-12-05 09:08:38 +03:00
Roman Khimov
c034aae378 core: fix wrong data being read in interops (part of #501)
When 74590551 introduced this code we had no proper caching layer, so there
were these strange fallbacks in the code. fc0031e5 should'd removed them, but
failed to do so, so do it now and fix processing of transactions that touch
storage for the same key (address) in the same block.
2019-12-04 19:51:57 +03:00
Vsevolod Brekelov
ff15a0acfd core: add trigger types as constants. Closes #509 2019-12-04 12:27:04 +03:00
Roman Khimov
31add423a8 core: replace open-coded PUSHBYTES64 with the one from opcode 2019-12-03 18:23:46 +03:00
Roman Khimov
138e125646 *: remove duplicate functions producing verification script
Drop wif.GetVerificationScript(), drop
smartcontract.CreateSignatureRedeemScript(), add GetVerificationScript()
directly to the PublicKey and use it everywhere.
2019-12-03 18:23:46 +03:00
Roman Khimov
8d4dd2d2e1 vm: move opcodes into their own package
This allows easier reuse of opcodes and in some cases allows to eliminate
dependencies on the whole vm package, like in compiler that only needs opcodes
and doesn't care about VM for any other purpose.

And yes, they're opcodes because an instruction is a whole thing with
operands, that's what context.Next() returns.
2019-12-03 18:22:14 +03:00
Roman Khimov
f48228ef7d
Merge pull request #467 from nspcc-dev/errcheck_297
This patchset closes #297 and #457.
2019-12-03 15:06:11 +03:00
Vsevolod Brekelov
03ff2976ed io: refactoring for using WriteVarBytes instead of WriteLE
goal is to be consistent with C# implementation.
For writing []byte WriteBytes used and for byte - WriteVarByte.
2019-12-03 13:49:33 +03:00
Roman Khimov
1c89c192ac core: optimize some accesses to unsortedTxn in mempool
These don't need a full lock, they only read things from maps.
2019-12-02 22:39:43 +03:00
Roman Khimov
0cd3493fa5 core: fix potential locking problem in mempool
I think it should fix this issue mentioned in the #526:

INFO[1456] blockchain persist completed                  blockHeight=63480 headerHeight=1810000 persistedBlocks=1 persistedKeys=4 took=740.7113ms
fatal error: concurrent map read and map write

goroutine 322 [running]:
runtime.throw(0xc8a6dc, 0x21)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc003473958 sp=0xc003473928 pc=0x42e282
runtime.mapaccess2(0xb706a0, 0xc0001893b0, 0xc0034739c8, 0xc0028704e0, 0x3)
        /usr/local/go/src/runtime/map.go:470 +0x278 fp=0xc0034739a0 sp=0xc003473958 pc=0x40dc08
github.com/CityOfZion/neo-go/pkg/core.MemPool.ContainsKey(0xc0001d0d20, 0xc0001893b0, 0xc0001893e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /neo-go/pkg/core/mem_pool.go:92 +0xcb fp=0xc003473a30 sp=0xc0034739a0 pc=0x9326db
github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).HasTransaction(0xc0001ca8c0, 0x49f0b45d430e441b, 0x553db79b7072821c, 0x28969518de11976, 0xba5100efddbe79d4, 0xc003473cd0)
        /neo-go/pkg/core/blockchain.go:803 +0xa1 fp=0xc003473b68 sp=0xc003473a30 pc=0x914b11
github.com/CityOfZion/neo-go/pkg/core.Blockchainer.HasTransaction-fm(0x49f0b45d430e441b, 0x553db79b7072821c, 0x28969518de11976, 0xba5100efddbe79d4, 0xc005a5d388)
        /neo-go/pkg/core/blockchainer.go:28 +0x46 fp=0xc003473ba8 sp=0xc003473b68 pc=0x997326
github.com/CityOfZion/neo-go/pkg/network.(*Server).handleInvCmd(0xc00018f680, 0xd9d980, 0xc00025e190, 0xc005a5d380, 0x0, 0x0)
        /neo-go/pkg/network/server.go:401 +0x3bb fp=0xc003473d38 sp=0xc003473ba8 pc=0x9924cb
github.com/CityOfZion/neo-go/pkg/network.(*Server).handleMessage(0xc00018f680, 0xd9d980, 0xc00025e190, 0xc007a0d050, 0x0, 0x0)
        /neo-go/pkg/network/server.go:582 +0x1ae fp=0xc003473da0 sp=0xc003473d38 pc=0x993bbe
github.com/CityOfZion/neo-go/pkg/network.(*TCPTransport).handleConn(0xc000228420, 0xd9b880, 0xc0001b6f00)
        /neo-go/pkg/network/tcp_transport.go:93 +0x202 fp=0xc003473fc8 sp=0xc003473da0 pc=0x996672
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc003473fd0 sp=0xc003473fc8 pc=0x45b3e1
created by github.com/CityOfZion/neo-go/pkg/network.(*TCPTransport).Dial
        /neo-go/pkg/network/tcp_transport.go:36 +0xb4

The problem is that we're modifying `unsortedTxn` under a reader lock.
2019-12-02 22:36:59 +03:00
Roman Khimov
bc5beb438f *: gofmt
Somehow these crept into the repository.
2019-12-02 10:10:50 +03:00
Roman Khimov
2c0ed62680 core: fix blockchain init with (2000*N + 1) header hashes in DB
Stored header count includes block number zero, so it's (height + 1).
2019-11-29 16:26:44 +03:00
Roman Khimov
2d41450ac9
Merge pull request #507 from nspcc-dev/dbft
network: plug in dBFT library
2019-11-27 15:22:40 +03:00
Roman Khimov
c5e2b3ce73
Merge pull request #523 from nspcc-dev/fix-interim-block-changes-store
Fix interim block changes store
2019-11-27 14:51:29 +03:00
Vsevolod
1a9e98caf3
Merge pull request #522 from nspcc-dev/fix-invalid-tx-type-handling
Fix invalid tx type handling
2019-11-27 13:40:52 +03:00
Roman Khimov
467c9c146d core: restore intermediate block-level store
Commit c80ee952a1 removed temporary store used
to contain changes of the block being processed. It's wrong in that the block
changes should be applied to the database in a single transaction so that
there wouldn't be any intermediate state observed from the outside (which is
possible now). Also, this made changes commiting persist them to the
underlying store effectively making our persist loop a no-op (and not
producing `persist completed` log lines that we love so much).
2019-11-27 13:28:13 +03:00
Roman Khimov
bb765ccfa7 core: gofmt 2019-11-27 13:27:47 +03:00
Roman Khimov
65bf0d01d5 transaction: don't hide decoding error by logging
Unknown transaction types are wrong and should be treated as errors.
2019-11-27 13:12:04 +03:00
Evgenii Stratonikov
fdd5276d3e network: plug in dBFT library 2019-11-27 10:57:22 +03:00
Evgenii Stratonikov
af61571ba0 core: do not return error in block.getHashableData 2019-11-26 15:27:37 +03:00
Roman Khimov
eb84ae49da
Merge pull request #508 from nspcc-dev/feature/getvalidators_420
core: Implement getvalidators interop
2019-11-22 18:15:08 +03:00
Vsevolod Brekelov
c03de88b41 core: unit tests blockchain state 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
d28f2dce05 core: unit test validator state 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
419b031e96 core: add weighted function for validators 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
672f314be0 core: nit refactoring and add godoc 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
8ae88593dd core: blockchain fmt 2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
3c4b0e955d core: create blockchain state that represents state of the bc
move some util methods to blockchain state
2019-11-21 15:41:49 +03:00
Vsevolod Brekelov
c80ee952a1 core/crypto: add validators and interop for validators
add processing of validators while block persist;
add validator structure with decoding/encoding;
add validator get from store;
add EnrollmentTX and StateTX processing;
add pubkey decode bytes, unique and contains functions;
2019-11-21 15:23:35 +03:00
Roman Khimov
34e2122e58 core: only check tx against mempool if it's not in the block
Fixes failure to process transaction from the block when it was relayed
initially:

WARN[0788] blockQueue: failed adding block into the blockchain  blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
WARN[0790] blockQueue: failed adding block into the blockchain  blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
WARN[0790] blockQueue: failed adding block into the blockchain  blockHeight=7270 error="transaction 35088916403e5cf2152e16c3bc6e0fba20c955fba38543b9fa5c50a3d3a4ace5 failed to verify: invalid transaction due to conflicts with the memory pool" nextIndex=7271
2019-11-20 16:12:40 +03:00
Roman Khimov
27a57e1a2d transaction: allow system fee specification in NewInvocationTX
It's not possible to create any deployment TX without it.
2019-11-20 16:12:40 +03:00
Roman Khimov
29882b076c rpc: remove duplicating definition of UTXO
Port sorting methods to core.
2019-11-19 17:35:04 +03:00
Roman Khimov
d2bdae99e4
Merge pull request #492 from nspcc-dev/feature/interop_neo_tests1
core: add unit tests for interops_neo
2019-11-18 19:39:05 +03:00
Vsevolod Brekelov
d57b6e1091 core: add unit tests for interops_neo 2019-11-18 19:31:27 +03:00
Roman Khimov
5841d3931e rpc: implement getunspents method, fix #473 2019-11-18 11:30:09 +03:00
Roman Khimov
e4868cd429 core: implement EncodeBinary on value
Fixes panic:
panic: core.NotificationEvent does not have EncodeBinary(*BinWriter)

goroutine 97 [running]:
github.com/CityOfZion/neo-go/pkg/io.(*BinWriter).WriteArray(0xc004f7dda0, 0xabf820, 0xc004f7ddc0)
        /home/rik/dev/neo-go/pkg/io/binaryWriter.go:45 +0x4b7
github.com/CityOfZion/neo-go/pkg/core.(*AppExecResult).EncodeBinary(0xc000635498, 0xc004f7dda0)
        /home/rik/dev/neo-go/pkg/core/notification_event.go:73 +0x9c
github.com/CityOfZion/neo-go/pkg/core.putAppExecResultIntoStore(0xcf0020, 0xc001aafec0, 0xc000635498, 0xc004f81940, 0x40)
        /home/rik/dev/neo-go/pkg/core/notification_event.go:33 +0xab
github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).storeBlock(0xc000118000, 0xc006f89ea0, 0xc006f89ea0, 0x0)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:534 +0x2539
github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).AddBlock(0xc000118000, 0xc006f89ea0, 0xc004f368b0, 0x1)
        /home/rik/dev/neo-go/pkg/core/blockchain.go:256 +0x91
github.com/CityOfZion/neo-go/pkg/network.(*blockQueue).run(0xc0048f4b80)
        /home/rik/dev/neo-go/pkg/network/blockqueue.go:39 +0x14c
created by github.com/CityOfZion/neo-go/pkg/network.(*Server).Start
        /home/rik/dev/neo-go/pkg/network/server.go:133 +0x272
2019-11-15 22:05:09 +03:00
Roman Khimov
1a5731e005 core: rework balance accounting
Store all unspents instead of balance values. Bump store version as this is an
incompatible change.
2019-11-15 22:05:09 +03:00
Roman Khimov
a16c2c3825 core: add NEP5 transfer tracking stub
See #498.
2019-11-15 18:04:59 +03:00
Roman Khimov
01082a8988 core: fix nondeterministic txGetReferences() behavior
Ranging over map is nondeterministic and contracts may be unprepared for
that. Fixes #454.
2019-11-15 18:04:59 +03:00
Roman Khimov
5e8122bfac core: add contract hash into the error output for checkStorageContext()
Makes debugging things easier.
2019-11-15 18:04:59 +03:00
Roman Khimov
5ce269c035 core: change notify implementation to save notifications
Fixes #453 and makes it possible to refer to these notifications later.
2019-11-15 18:04:59 +03:00
Evgenii Stratonikov
1a08ad19aa sc: implement io.Serializable for ParamType 2019-11-14 14:20:38 +03:00
Evgenii Stratonikov
2be18f91df util: implement io.Serializable for Uint256 2019-11-14 14:20:38 +03:00
Evgenii Stratonikov
b16e56a47b io: refactor BinReader.ReadArray()
Make it accept arbitrary slice pointer.
2019-11-14 14:19:58 +03:00
Evgenii Stratonikov
ad9091d13d io: implement generic array (de-)serialization
It is done through reflection and panics
in every unexpected situation.
2019-11-13 17:27:23 +03:00
Roman Khimov
394346fc26 core: add proper claim transactions processing 2019-11-11 15:35:57 +03:00
Roman Khimov
b05754deac core: add Close() to blockchainer, implement it to properly close chain
Before it the deferred function in Run() was actually never able to properly
close the Store, so we weren't synching the latest state to the disk.
2019-11-08 12:19:54 +03:00
Vsevolod
f686069f37
Merge pull request #481 from nspcc-dev/core-fix-init-corner-cases
Fix init corner cases
2019-11-06 18:36:40 +03:00
Roman Khimov
c567307748 core: fix potential issue with header list init
If we're to receive some 500 headers (less than `headerBatchCount`) and quit
before receiving more of them we end up with clean `bc.headerList` that will
be inited going backwards to the `targetHash`, but code path doesn't add add
the `targetHash` itself which it should do in this particular case, otherwise
we end with no genesis block hash in the list.
2019-11-06 18:13:45 +03:00
Roman Khimov
b5199625cd core: init genesis block header if there are no other headers in DB
Fixes crash when restarted after the DB initialization and no blocks written
into the DB.
2019-11-06 17:58:19 +03:00
Roman Khimov
c16c2bf102 core: store current header reference on clean init
Missing it the following line could fail on subsequent restarts:

        currHeaderHeight, currHeaderHash, err := storage.CurrentHeaderHeight(bc.store)

if the node was stopped before any headers had been received.
2019-11-06 17:56:06 +03:00
Vsevolod Brekelov
d799c98cfe io: move common function and add unit tests for it 2019-11-06 17:12:33 +03:00
Roman Khimov
9f7625d699 core: don't do useless DB reads if nothing was persisted 2019-11-06 16:10:37 +03:00
Roman Khimov
90d4299f0b core: gofmt blockchain.go 2019-11-06 16:09:45 +03:00
ed758458d5 vm: implement serialization interops in core package 2019-11-05 17:10:52 +03:00
Roman Khimov
2f6e678a19
Merge pull request #463 from nspcc-dev/smartcontract-fixes
Smartcontract RPC fixes
2019-10-29 20:54:46 +03:00
Vsevolod Brekelov
d374175170 monitoring: add prometheus monitoring
add init metrics service which uses prometheus;
add configuration for metrics service;
add monitoring metrics for blockchain,rpc,server;
2019-10-29 20:51:17 +03:00
Roman Khimov
ebc1ba4f38 rpc/core: implement invokescript method, fix #348
Extend Blockchainer with one more method to spawn a VM for test runs and use
it to run scripts. Gas consumption is not counted or limited in any way at the
moment (see #424).
2019-10-29 18:31:39 +03:00
Roman Khimov
a568740f56 core: deduplicate a part of VM setup into spawnVMWithInterops() 2019-10-29 17:10:50 +03:00
Roman Khimov
99bbad331d core: add issue tx processing
Count available asset quantity, match C# code.
2019-10-25 17:20:12 +03:00
Roman Khimov
45cac07643 core: remove transactions from MemPool when they're stored in block
Extend MemPool with Remove() and use it. Fixes #446.
2019-10-24 13:12:02 +03:00
Roman Khimov
184d8a0180 core: fix missing unlock in MemPool TryAdd()
Fixes deadlock on subsequent MemPool operations.
2019-10-24 13:12:01 +03:00
Roman Khimov
8d0ed9259c transaction: fix invocation TX consistency in NewInvocationTX()
NewInvocationTX() returned a version number one transaction that actually
failed to pass that version down to the invocation data which lead to
serialization/deserialization inconsistency.
2019-10-23 14:50:24 +03:00
Roman Khimov
1250bf9579
Merge pull request #443 from nspcc-dev/spellcheck
spellcheck and comments fix
2019-10-22 18:01:40 +03:00
Vsevolod Brekelov
8ee421db14 fix spelling and godoc comments 2019-10-22 17:56:03 +03:00
Roman Khimov
3cbb699eb7
Merge pull request #426 from nspcc-dev/logger_247
Change fmt.Println to log, close #247.
2019-10-22 14:41:30 +03:00
Vsevolod Brekelov
e2bfff8666 vm: removed mute mode and pushed logging to upper lvl
VM should be responsible for code execution and in case anyone interested in additional logging or errors they could handle them like we do it iin cli.
2019-10-22 13:44:14 +03:00
Roman Khimov
f1750d117b
Merge pull request #440 from nspcc-dev/db-dump-restore
DB dump and restore, fixes #436.
2019-10-21 15:42:05 +03:00
Roman Khimov
07c2105aa5 core: log values from the store in persist()
We're about stored values here, so print those, which avoids blocking in
bc.HeaderHeight() and removes duplication between blockHeight and
persistedHeight. Fixes saving the blockchain on exit (deferred function in
Run() blocked in persist()).

Test modification was required because storeBlocks() doesn't actually save
headers and thus TestGetTransaction started to fail on persist().
2019-10-21 14:18:09 +03:00
Roman Khimov
70407f0c19 core: remove unused context parameter from persist() 2019-10-21 14:18:09 +03:00
Vsevolod Brekelov
063fe5be9d unspentcoinstate: fix branch never executed 2019-10-21 12:40:11 +03:00
Roman Khimov
b533dfceba core: don't panic on init when there are less than 2000 hashes
If you're to sync less than 2000 headers no batched header key-value is
gonna be written into the DB and init() would panic because
bc.headerList.Len() would return 0. Use genesis block as a target in this
case.
2019-10-21 08:37:01 +03:00
Vsevolod Brekelov
fe39c565b9 codestyle: redundant type conversion 2019-10-18 18:39:31 +03:00
Vsevolod Brekelov
1afdb895a1 codestyle: declare empty slice
https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices
2019-10-18 18:38:33 +03:00
Vsevolod Brekelov
ced2022273 storage: change println error for botldb to log 2019-10-17 18:23:49 +03:00
Roman Khimov
3fc2bf5452 *: fix some misspellings
Goreport:
   neo-go/pkg/core/contract_state_test.go
        Line 21: warning: "Contracto" is a misspelling of "Contraction" (misspell)
        Line 64: warning: "Contracto" is a misspelling of "Contraction" (misspell)

   neo-go/pkg/core/interop_neo.go
        Line 420: warning: "succeedes" is a misspelling of "succeeds" (misspell)

   neo-go/pkg/network/discovery.go
        Line 118: warning: "succeded" is a misspelling of "succeeded" (misspell)
        Line 128: warning: "successfuly" is a misspelling of "successfully" (misspell)
2019-10-17 12:30:24 +03:00
Roman Khimov
1b5ea67737 core: fix ineffassign suggestion
Goreport:
  Line 467: warning: ineffectual assignment to persisted (ineffassign)
2019-10-17 12:27:15 +03:00
Roman Khimov
288000a8af storage: drop useless nil assignment in leveldb code
golint suggests:
   Line 24: warning: should drop = nil from declaration of var opts; it is the zero value (golint)
2019-10-17 12:18:16 +03:00
Roman Khimov
2245fedbb1 core: deduplicate state commit methods, use interim MemCachedStore
commit methods duplicated putSmthIntoStore functions, but have MemCachedStore
now that can easily substitute for a Batch, especially given that interop
needs something like that for its storage purposes anyway.
2019-10-16 17:33:45 +03:00
Roman Khimov
fc0031e5aa core: move write caching layer into MemCacheStore
Simplify Blockchain and associated functions, deduplicate code, fix Get() and
Seek() implementations.
2019-10-16 17:33:45 +03:00
Roman Khimov
8926cbe368 core: fix potential data race in logging code
Spotted in CircleCI build with Go 1.12:

WARNING: DATA RACE
Write at 0x00c00011095c by goroutine 88:
  sync/atomic.SwapInt32()
      /usr/local/go/src/runtime/race_amd64.s:249 +0xb
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).persist()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:483 +0x172
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run.func2()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:191 +0x50

Previous read at 0x00c00011095c by goroutine 64:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).persist()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:492 +0x4be
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run.func2()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:191 +0x50

Goroutine 88 (running) created at:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:190 +0x264

Goroutine 64 (finished) created at:
  github.com/CityOfZion/neo-go/pkg/core.(*Blockchain).Run()
      /go/src/github.com/CityOfZion/neo-go/pkg/core/blockchain.go:190 +0x264
2019-10-15 18:58:22 +03:00
Roman Khimov
a6610ba082 core: verify blocks, fix #12
This adds the following verifications:
 * merkleroot check
 * index check
 * timestamp check
 * witnesses verification

VerifyWitnesses is also renamed to verifyTxWitnesses here to not confuse it
with verifyBlockWitnesse and to hide it from external access (no users at the
moment).
2019-10-15 18:58:17 +03:00
Roman Khimov
03c20f1876 core: make ContractState receiver names consistent
Linter isn't happy with our recent changes:

pkg/core/contract_state.go:109:1: receiver name cs should be consistent with previous receiver name a for ContractState
pkg/core/contract_state.go:114:1: receiver name cs should be consistent with previous receiver name a for ContractState
pkg/core/contract_state.go:119:1: receiver name cs should be consistent with previous receiver name a for ContractState

But actually `a` here most probably is a copy-paste from AssetState methods,
so fit the old code to match the new one.
2019-10-15 12:56:25 +03:00
Roman Khimov
aec6a5f029 core/config: make block and transaction verification configurable
Enable transaction verification for privnets and tests, testnet can't
successfuly verify block number 316711 with it enabled and mainnet stops at
105829.
2019-10-15 12:56:25 +03:00
Roman Khimov
56dcff2894 core: invoke contracts for Invocation TXes
They can have some side-effects that future invocations rely on.
2019-10-15 12:56:25 +03:00
Roman Khimov
74590551c4 core: add some interops
This also changes Verify to VerifyTx and VerifyWitnesses, because there is a
need to pass a block for some interop functions.
2019-10-15 12:56:25 +03:00
Roman Khimov
667f346c04 core: improve error message in AddBlock test 2019-10-15 12:56:25 +03:00
Roman Khimov
8266a5ce19 core: export GetContractState/GetScriptHashesForVerifying via Blockchainer
These are gonna be used by interops and are also useful in general.
2019-10-15 12:56:25 +03:00
Roman Khimov
19fd7f844e core: add GetUnspentCoinState() for future interops 2019-10-15 12:56:25 +03:00
Roman Khimov
bfddf9b3f6 core: implement StorageItem for future interops 2019-10-15 12:56:25 +03:00
Roman Khimov
2156d5db73 core: add put/delete functions for ContractState
These are gonna be used by interops.
2019-10-15 12:56:25 +03:00
Roman Khimov
58d9b79fe3 core: avoid duplicates in GetScriptHashesForVerifyingClaim()
As they can break VerifyWitnesses().
2019-10-15 12:56:25 +03:00
Roman Khimov
35824728eb core: add putAssetStateIntoStore()
Will be used by interops.
2019-10-15 12:56:25 +03:00
Roman Khimov
96934cfeab core: return error from verifyOutputs()
Make debugging easier.
2019-10-15 12:56:25 +03:00
Roman Khimov
ab4ff79e02 core: fix typo in GetTransactionResults() 2019-10-15 12:56:25 +03:00
Roman Khimov
cffe8d0ee2 core: return error from verifyResults()
Make it easier to find out what the real problem is.
2019-10-15 12:56:25 +03:00
Roman Khimov
16bc5296cb block: return error from Verify
Don't hide real problem behind the bool value. Makes it easier to identify
problems when looking at log messages.
2019-10-15 12:56:25 +03:00
Roman Khimov
258f397b9a core: append transactions to the block in GetBlock()
We want to get a full block, so it has to have transactions
inside. Unfortunately our tests were used to this wrong behavior and utilized
completely bogus transactions without data that couldn't be persisted, so fix
that also.
2019-10-15 12:56:25 +03:00
Roman Khimov
238c590ddb core: fix contract state's Properties to use PropertyState
PublishTX only had one of these flags, but newer contracts (created via the
interop function) can have more and these flags are aggregated into one field
that uses PropertyState enumeration (it's used to publish contract, so
supposedly it's also a nice choice for contract state storage).
2019-10-15 12:56:25 +03:00
Roman Khimov
78861485b6 storage: simplify MemoryBatch
It's used a lot and it looks a lot like MemoryStore, it just needs not to
return errors from Put and Delete, so make it use MemoryStore internally with
adjusted interface.
2019-10-15 12:56:25 +03:00
Roman Khimov
13bf2618ef storage: improve PutBatch for MemoryStore
Make it look more like a real transaction, put/delete things with a single
lock. Make a copy of value in Put also, just for safety purposes, no one knows
how this value slice can be used after the Put.
2019-10-15 12:56:25 +03:00
Roman Khimov
3ada92944a storage: drop Len from the Batch interface
It's almost meaningless now and we can easily live without it.
2019-10-15 12:56:25 +03:00
Roman Khimov
e111892653 storage: redo DB testing
Make generic tests for all Store implementations, deduplicate tests. Implement
Delete() tests, test Seek() better, add LevelDB tests (finally!).
2019-10-15 12:56:25 +03:00
Roman Khimov
48b6a427cf storage: add Delete method for Batch and Store
It's gonna be used by Storage and Contract interops, both can delete their
data.
2019-10-07 19:40:11 +03:00
Roman Khimov
add9368e9d storage: use strings as keys for memory batch
Using pointers is just plain wrong here, because the batch can be updated with
newer values for the same keys.

Fixes Seek() to use HasPrefix also because this is the intended behavior.
2019-10-07 17:05:53 +03:00
Roman Khimov
1bf232ad50 vm: introduce TryBool() for Element and use it in VerifyWitnesses
Script can return non-bool results that can still be converted to bool
according to the usual VM rules. Unfortunately Bool() panics if this
conversion fails which is OK for things done in vm.execute(), but certainly
not for VerifyWitnesses(), thus there is a need for TryBool() that will just
return an error in this case.
2019-10-04 16:13:39 +03:00
Roman Khimov
ceca9cdb67 core/vm: implement contract storage and script retrieval
Fixes script invocations via the APPCALL instruction. Adjust contract state
field types accordingly.
2019-10-04 16:13:39 +03:00
Roman Khimov
b7c1efe7e3 core: fix References() result key type
If the block references two ouputs in some other transaction the code failed
to verify it because of key collision. C# code implements it properly by using
full CoinReference type as a key, so let's do it in a similar fashion.
2019-10-01 13:41:26 +03:00
Roman Khimov
8537700b7b core: sort hashes and witnesses in VerifyWitnesses()
Fixes failure to verify some multi-witnesses transactions in Testnet chain. C#
code contains similar logic.
2019-10-01 13:41:26 +03:00
Roman Khimov
dd0dd2d031 core: add claim-specific GetScriptHashesForVerifying()
Claim transactions have different logic in C# node, so we need to
implement it too. It's not the most elegant way to fix it, but let's make it
work first and then refactor if and where needed. Fixes verification of Claim
transactions.
2019-10-01 13:41:26 +03:00
Roman Khimov
854fb187a3 core: verify transactions in AddBlock() 2019-10-01 13:41:26 +03:00
Roman Khimov
1095abb04c core: fix default asset expiration to match C# code
Fixes bogus verification failures for non-native assets.
2019-10-01 13:41:26 +03:00
Roman Khimov
832c56f97d core: fix DutyFlag check in GetScriptHashesForVerifying()
It's a flag (used by Share and Invoice asset types), so it should be checked
like a flag, the same way C# node does.
2019-10-01 13:41:26 +03:00
Roman Khimov
4ae18e8ffc block: check for Transaction length before messing with txes
Fixes panic two lines below. Blocks without transactions are invalid by
definition, so there is a need to adjust tests accordingly.
2019-10-01 13:41:26 +03:00
Roman Khimov
7779ad6c28 storage: always return ErrKeyNotFound when key is not found
It was true for MemoryStore and BoltDB, but not for LevelDB and Redis.
2019-09-27 15:42:36 +03:00
Roman Khimov
5ff1399d54 storage: unexport MemoryStore mutex
It shouldn't be available from outside.
2019-09-27 15:42:36 +03:00
Roman Khimov
3c40a53c4d core: no need to Close() Blockchain in tests
It's a Store method actually and for every Blockchain that uses Run() the
Close() will be handled automatically on exit because of context magic.
2019-09-27 15:42:35 +03:00
Roman Khimov
0dbdbb9c2a storage: remove RedisBatch in favor of MemoryBatch
They're also almost the same.
2019-09-27 15:42:35 +03:00
Roman Khimov
af557cea19 storage: deduplicate BoltDBBatch/MemoryBatch
BoltDB doesn't have internal batching mechanism, thus we have a substitute for
it, but this substitute is absolutely identical to MemoryBatch, so it's better
to unify them and import ac5d2f94d3 fix into the
MemoryBatch.
2019-09-27 15:42:35 +03:00
Roman Khimov
8bf37f45fc storage: remove impossible error branch from MemoryStore
MemoryStore is always successful, so this branch makes no sense.
2019-09-27 15:42:35 +03:00
Roman Khimov
4a732d97ac storage: add comments about errors from MemoryStore
It never returns errors for some operations and this knowledge can be used by
other code.
2019-09-27 15:42:35 +03:00
Roman Khimov
920d7c610c core: remove blockCache, use MemoryStore, redesign persist()
Commit 578ac414d4 was wrong in that it saved
only a part of the block, so depending on how you use blockchain, you may
still see that the block was not really processed properly. To really fix it
this commit introduces intermediate storage layer in form of memStore, which
actually is a MemoryStore that supports full Store API (thus easily fitting
into the existing code) and one extension that allows it to flush its data to
some other Store.

It also changes AddBlock() semantics in that it only accepts now successive
blocks, but when it does it guarantees that they're properly added into the
Blockchain and can be referred to in any way. Pending block queing is now
moved into the server (see 8c0c055ac657813fe3ed10257bce199e9527d5ed).

So the only thing done with persist() now is just a move from memStore to
Store which probably should've always been the case (notice also that
previously headers and some other metadata was written into the Store
bypassing caching/batching mechanism thus leading to some inefficiency).
2019-09-27 15:42:35 +03:00
Roman Khimov
c531dc0bde network: add block queue
This one will replace blockCache in Blockchain itself as it can and should be
external from it. The idea is that we only feed successive blocks into the
Blockchain and it only stores valid proper Blockchain and nothing else.
2019-09-27 13:00:09 +03:00
Roman Khimov
903173c991 core: fix deadlock in headers list init
This probably is a problem since adc880d323,
because AddHeaders() uses headersOp channel and its reader is only ran in
Run().
2019-09-25 17:52:46 +03:00
Roman Khimov
578ac414d4 core: make unpersisted blocks/txs available in Blockchain
This changes the Blockchain to also return unpersisted (theoretically, verified
in the AddBlock!) blocks and transactions, making Add/Get interfaces
symmetrical. It allows to turn Persist into internal method again and makes it
possible to enable transaction check in GetBlock(), thus fixing #366.
2019-09-25 17:46:28 +03:00
Roman Khimov
8a457ef540 transaction: add a check for createHash() invocation
It should work in general.
2019-09-24 13:01:54 +03:00
Roman Khimov
679e01bd0f core: implement witness verification logic
Fix #368.
2019-09-24 13:01:54 +03:00
Roman Khimov
ac5d2f94d3 storage: fix BoltDB batched Put()
It must copy both the value and the key because they can be reused for other
purposes between Put() and PutBatch(). This actually happens with values in
headers processing, leading to wrong data being written into the DB.

Extend the batch test to check for that.
2019-09-23 09:27:18 +03:00
Roman Khimov
4b4bac675b core: don't print persisting errors twice
As they're already printed in the calling goroutine. And they're alse not
always useful when printed by Persist().
2019-09-22 20:08:15 +03:00
Roman Khimov
102c926ef3 core: don't print useless persist messages with nil errors
Errors should be printed only when there are errors.
2019-09-22 20:07:49 +03:00
Roman Khimov
5cddfe071b
Merge pull request #403 from nspcc-dev/fix_rpctest
Fix rpctest, fix #353, fix #305.
2019-09-18 23:14:37 +03:00