Commit graph

830 commits

Author SHA1 Message Date
Roman Khimov
565c1940e3
Merge pull request #587 from nspcc-dev/feature/zap
Closes #284.
2020-01-10 14:50:09 +03:00
Evgenii Stratonikov
9dc5571327 core,rpc: close Blockchain in tests
If blockchain is not closed, logging in defer can occur
after test has finished, which will lead to a panic with
"Log in goroutine after Test* has completed".
2020-01-10 11:47:56 +03:00
Evgenii Stratonikov
dc65684708 consensus: replace SugaredLogger with Logger
SugaredLogger was used to make output look more similar to logrus.
This is no longer needed.
2020-01-10 11:14:29 +03:00
Evgenii Stratonikov
9b8b77c9ea network: return error if header message is too big
Big messages can still be processed but only first
2000 headers will be used.
2020-01-10 11:14:29 +03:00
Evgenii Stratonikov
637c99eda7 core: simplify newInteropContext 2020-01-10 11:14:29 +03:00
Evgenii Stratonikov
45a4524054 rpc: remove EnableCORSWorkaround from Request 2020-01-10 11:14:29 +03:00
Evgenii Stratonikov
289cb1c1d9 rpc: use zap.Logger 2020-01-10 11:14:29 +03:00
Evgenii Stratonikov
aecdf470e7 cli,pkg: use zap.Logger 2020-01-10 11:14:27 +03:00
Roman Khimov
e96b4bc82e wallet: fix TestNewAccount
It duplicated TestNewFromWif.
2020-01-09 18:40:30 +03:00
Roman Khimov
9bb68d7025 wallet: make DecryptAccount a method of Account
Otherwise it almost duplicates keys.NEP2Decrypt().
2020-01-09 18:40:30 +03:00
Roman Khimov
951ee383e9 keys: improve NEP2 testing 2020-01-09 18:06:03 +03:00
Roman Khimov
b5b05a969c keys: make NEP2Decrypt return a PrivateKey rather than WIF
There is no point in encoding the output of this function in a WIF format,
most of the users actually want the real key and those who need a WIF can
easily get if from the key (and it's simpler than getting the key from the
WIF).

It also fixes a severe bug in NEP2Decrypt, base58 decoding errors were not
processed correctly.
2020-01-09 18:05:14 +03:00
Evgenii Stratonikov
9c79684516 storage: panic on error in boltdb.Seek
Error in Seek means something is terribly wrong (e.g. db was not opened) and
error drop is not the right thing to do, because caller
will continue working with the wrong view.
2020-01-09 15:17:35 +03:00
Evgenii Stratonikov
8d1f564411 consensus: add tests for getTx 2019-12-27 14:12:58 +03:00
Evgenii Stratonikov
d803dffca8 consensus: return nil interface from getTx 2019-12-27 13:54:11 +03:00
Roman Khimov
c306d2f07c
Merge pull request #581 from nspcc-dev/feat/storage_find
core: implement Neo.Storage.Find interop, closes #422.
2019-12-26 15:46:46 +03:00
Evgenii Stratonikov
48cf4f4a84 core: implement Neo.Storage.Find interop 2019-12-26 15:21:41 +03:00
Evgenii Stratonikov
7b5c47e7f5 vm: add NewMapIterator() 2019-12-26 14:34:15 +03:00
Roman Khimov
f8887b63ba
Merge pull request #578 from nspcc-dev/fix/sign
consensus: verify payloads correctly, closes #555.
2019-12-26 13:07:40 +03:00
Evgenii Stratonikov
64d24d8ddd consensus: verify payloads correctly 2019-12-26 10:49:56 +03:00
Evgenii Stratonikov
714c466c2c core: add ScriptFromWitness function
Extracting verification script from witness became a common task.
This commit adds such a possibility.
2019-12-26 10:49:56 +03:00
Roman Khimov
234d94d27e network: implement getblocks command
Fixes #577, tested with C# nodes connecting to neo-go privnet.
2019-12-25 19:46:57 +03:00
Roman Khimov
9145855d2c
Merge pull request #579 from nspcc-dev/refactor-crypto
This moves some functionality into micro-packages, improves testing,
unexports some code and fixes bugs along the way.
2019-12-25 18:12:50 +03:00
Roman Khimov
604b2c8240 keys: add a successful test for 04-encoded public key 2019-12-25 18:06:25 +03:00
Roman Khimov
fa0537758f keys: add a test for X/Y > P decoding case 2019-12-25 18:01:54 +03:00
Roman Khimov
6c471ecd98 keys: move IsOnCurve decoding check, add a test for it
This check only makes sense for 04-encoded points, because 02 and 03 derive Y
from X and they're on the curve by definition.
2019-12-25 18:00:25 +03:00
Roman Khimov
5ac8cae221 keys: fix bad compressed public key decoding, add a test
The error was not propagated properly here.
2019-12-25 17:44:30 +03:00
Roman Khimov
b246653f62 address: rename functions as per #579 comments
Make them more clear to understand.
2019-12-25 17:34:18 +03:00
Roman Khimov
53e6fbbbf9 base58: add a test for a025b9c42d
As noted by @fyrchik in #579 comments.
2019-12-25 17:33:48 +03:00
Roman Khimov
e25133a385 keys: reuse address package for address generation
Deduplicates code and makes prefix overridable.
2019-12-25 16:03:39 +03:00
Roman Khimov
b7702f3a2e address: improve testing, make it 100% covered 2019-12-25 15:55:39 +03:00
Roman Khimov
c5ceb9128a address: check for prefix when decoding 2019-12-25 15:50:52 +03:00
Roman Khimov
89b6cbf795 address: make Prefix overridable
As it should be, it's specified in the configuration file (and it should be
treated as byte in the config)
2019-12-25 15:42:18 +03:00
Roman Khimov
e685e9bf9a address: move into its own package
Doesn't really belong to the crypto.
2019-12-25 15:22:02 +03:00
Roman Khimov
a025b9c42d base58: remove bogus code from CheckDecode
No other implementation does that and it looks weird, I see nothing meaninful
here.
2019-12-25 15:07:11 +03:00
Roman Khimov
369ac01a27 base58: move into its own package
It doesn't belong to crypto in any way other than it uses hash function
internally.
2019-12-25 15:05:54 +03:00
Roman Khimov
5dd8d29534 hash: improve merkle tree testing
Make it 100% covered.
2019-12-25 14:43:50 +03:00
Roman Khimov
ddad9ac9a3 hash: simplify merkle tree error handling
buildMerkleTree() is internal to the hash package and if anyone calls it with
`len(leaves) == 0` he deserves a panic. As it's the only error case in it, we
can remove error value return from this function and simplify NewMerkleTree().
2019-12-25 14:38:48 +03:00
Roman Khimov
ee28fb08f6 crypto: move merkle tree into the hash package
It's all about hashes, so it makes sense putting it there.
2019-12-25 11:28:59 +03:00
Roman Khimov
db5555bb15 crypto: internalize aes functions into the keys package
This is the only user of it and no one outside should care about these
details.
2019-12-25 11:25:05 +03:00
Roman Khimov
9cc0fca9d2
Merge pull request #574 from nspcc-dev/feature/multireturn
compiler: implement multiple return support, closes #562.
2019-12-25 08:46:38 +03:00
Vsevolod Brekelov
606616618f rpc: add ability to answer as C# with block as []byte 2019-12-25 01:44:37 +03:00
Vsevolod Brekelov
7f1844a39e transaction: add unmarshaling for TXType 2019-12-25 01:44:37 +03:00
Evgenii Stratonikov
f4571ba8cf compiler: implement multiple return support 2019-12-24 16:46:43 +03:00
Evgenii Stratonikov
5bc32b523a vm: implement Neo.Iterator.* interops 2019-12-24 10:21:47 +03:00
Evgenii Stratonikov
3ff7fd5262 vm: implement Neo.Enumerator.* interops 2019-12-24 10:21:46 +03:00
Roman Khimov
bf84e1f2fb core: cache top block
Turns out, our dApps use it a lot and we were going to the DB to get it which
is a useless waste of time. Technically we could also remove blockHeight here,
but not doing it at the moment as it's more involved.

It eliminates this time waste from the pprof graph, but doesn't change 1.4M ->
1.5M 100K mainnet block import test case in any noticeable way.
2019-12-23 19:18:12 +03:00
Roman Khimov
cc16dac0b4 vm: optimize script loading a bit
Preseed the scriptHash value when we already know it. Eliminates this time
waste from the pprof graph, but doesn't really change anything in the 1.4M ->
1.5M 100K mainnet blocks import test.
2019-12-23 18:02:39 +03:00
Roman Khimov
094c8474b7 compiler: move tests from vm/tests
These don't belong to VM as they compile some Go code and run it in a VM. One
may call them integration tests, but I prefer to attribute them to
compiler. Moving these tests into pkg/compiler also allows to properly count
the compiler coverage they add:

-ok     github.com/CityOfZion/neo-go/pkg/compiler       (cached)        coverage: 69.7% of statements
+ok     github.com/CityOfZion/neo-go/pkg/compiler       (cached)        coverage: 84.2% of statements

This change also fixes `contant` typo and removes fake packages exposed to the
public by moving foo/bar/foobar into the testdata directory.
2019-12-23 17:05:34 +03:00
Roman Khimov
deb76c5199 core: add negative non-interop test for all interops
Try to feed some garbage into these functions.
2019-12-23 16:14:43 +03:00
Roman Khimov
76d69ee5ee core: restore proper interops ordering 2019-12-23 16:14:43 +03:00
Roman Khimov
c287a9e93c core: implement Neo.Witness.GetVerificationScript interop 2019-12-23 16:14:43 +03:00
Roman Khimov
9d638d0fee core: implement Neo.InvocationTransaction.GetScript 2019-12-23 16:14:43 +03:00
Roman Khimov
33958be45f
Merge pull request #559 from nspcc-dev/feature/stack_limits
vm: don't refcount simple items

Improves 1.4M to 1.5M 100K mainnet block import test by ~4%.
2019-12-23 12:44:01 +03:00
Vsevolod Brekelov
d576aa2753 compiler: remove not used functions 2019-12-20 13:05:47 +03:00
Roman Khimov
79323cc10b
Merge pull request #567 from nspcc-dev/feature/codegen_add_mod
compiler: Add Mod token, closes #563
2019-12-19 15:32:18 +03:00
Vsevolod Brekelov
1458116567 vm: add tests 2019-12-19 15:23:14 +03:00
Vsevolod Brekelov
b68e9591aa compiler: add test for codegen 2019-12-19 15:23:14 +03:00
Vsevolod Brekelov
3c7ac7eb95 compiler: add mod 2019-12-19 15:23:14 +03:00
Roman Khimov
26ea4799c3
Merge pull request #565 from nspcc-dev/hashed-interop-callback
vm/core: add ID support for SYSCALL, redo interop registration
2019-12-19 15:02:53 +03:00
Roman Khimov
5f3a220b48
Merge pull request #566 from nspcc-dev/feature/varindex
compiler: implement assignment to a variable index
2019-12-19 13:40:32 +03:00
Roman Khimov
a7457d08a1 vm/core: add ID support for SYSCALL, redo interop registration
This solves two problems:
 * adds support for shortened SYSCALL form that uses IDs (similar to #434, but
   for NEO 2.0, supporting both forms), which is important for compatibility
   with C# node and mainnet chain that uses it from some height
 * reworks interop plugging to use callbacks rather than appending to the map,
   these map mangling functions are clearly visible in the VM profiling
   statistics and we want spawning a VM to be fast, so it makes sense
   optimizing it. This change moves most of the work to the init() phase
   making VM setup cheaper.

Caveats:
 * InteropNameToID accepts `[]byte` because that's the thing we have in
   SYSCALL processing and that's the most often usecase for it, it leads to
   some conversions in other places but that's acceptable because those are
   either tests or init()
 * three getInterop functions are: `getDefaultVMInterop`, `getSystemInterop`
   and `getNeoInterop`

Our 100K (1.4M->1.5M) block import time improves by ~4% with this change.
2019-12-19 13:35:42 +03:00
Evgenii Stratonikov
891a878af1 compiler: implement assignment to a variable index
Fixes #564.
2019-12-19 13:14:17 +03:00
Evgenii Stratonikov
f957af35d4 vm: do not store items of scalar types in map
As they do not contain any other items,
they can be only accounted via total size.
2019-12-18 11:12:19 +03:00
Roman Khimov
1a26548be8
Merge pull request #561 from nspcc-dev/fix-duping-and-and-tests
Fix duping and add tests.

C# node actually implements DUP in the same way we did, but it does create a
new element when accessing some particular value (like BigInt() or Bytes()) so
in the end this DUP implementation doesn't lead to any visible side-effects. In
our case I think it's more appropriate to fix the DUP (and its variants) itself
avoiding useless allocations in the VM.
2019-12-18 11:07:41 +03:00
Roman Khimov
10766fe813 vm: add tests for hashing instructions
These were cross-checked with the C# implementation.
2019-12-17 20:31:16 +03:00
Roman Khimov
21efcb012b vm: fix non-dupped items in PICKITEM
TestPICKITEMDupMap and TestPICKITEMDupArray were failing.
2019-12-17 20:26:30 +03:00
Roman Khimov
a6d60e387a vm: fix OVER and PICK to duplicate stack items
TestPICKDup and TestOVERDup failed without this.
2019-12-17 20:26:30 +03:00
Roman Khimov
60dfa05b19 vm: duplicate an item in Dup
TestDupByteArray and TestDupInt were failing before this patch.
2019-12-17 20:26:30 +03:00
Roman Khimov
c596a6b273 vm: fix INVERT behaviour for converted values
Tests added were failing before this change.
2019-12-17 20:26:30 +03:00
Evgenii Stratonikov
884779e501 *: goimports 2019-12-17 14:51:28 +03:00
Evgenii Stratonikov
014be31a0a network: fix link in pprof 2019-12-17 14:49:28 +03:00
Vsevolod
27a5825d42
Merge pull request #536 from nspcc-dev/feature/pprof
network: add Pprof metrics
2019-12-17 14:21:25 +03:00
Vsevolod Brekelov
0085831ec5 network: add Pprof metrics
Since we have some perf issues from time to time it is good to have pprof debugger. Disabled by default.
2019-12-17 14:13:07 +03:00
Roman Khimov
587cfc7c66 vm: optimize ROLL/ROT, refactor common code
Add `Roll` method to Stack that doesn't pop and push values and use it for
ROLL and ROT.

1.4M->1.5M 100K block import test before:
real    3m44,292s
user    5m43,494s
sys     0m34,741s

After:
real    3m40,449s
user    5m42,701s
sys     0m35,500s
2019-12-16 19:53:21 +03:00
Roman Khimov
2627628387 vm: optimize SWAP instruction, refactor common code
Add `Swap` method to the Stack and use it for both SWAP and XSWAP. Avoid
element popping and pushing (and associated accounting costs).

1.4M->1.5M 100K block import test before:
real    3m51,885s
user    5m54,744s
sys     0m38,444s

After:
real    3m44,292s
user    5m43,494s
sys     0m34,741s
2019-12-16 19:02:40 +03:00
Roman Khimov
7a5e995030 core: add MemPool Verify() test 2019-12-14 00:05:45 +03:00
Roman Khimov
5d68f88196 core: fix and speed up mempool Verify()
First of all, it was wrong, it was not checking for inputs really, it compared
tx hashes for some reason, second, when it did compare inputs it compared only
the PrevIndex part of them which is also wrong.

Also, there is absolutely no reason to go through GetVerifiedTransactions()
here, we don't need this copy of pointers and it can also be outdated by the
time we're to finish our check.

Before:
BenchmarkTXPerformanceTest-4
    5000            485506 ns/op           65886 B/op        409 allocs/op
ok      github.com/CityOfZion/neo-go/integration        3.212s

After:
enchmarkTXPerformanceTest-4
    5000            371104 ns/op           44367 B/op        408 allocs/op
ok      github.com/CityOfZion/neo-go/integration        2.712s
2019-12-13 23:23:33 +03:00
Roman Khimov
35f7f4137e
Merge pull request #556 from nspcc-dev/optimize-dao-and-things
Optimize dao and things, this set improves 1.4M->1.5M import of 100K blocks
by more than 43%.
2019-12-13 22:43:41 +03:00
Roman Khimov
e631e75718 core: optimize GetVerifiedTransactions()
This simple change improves our BenchmarkTXPerformanceTest by 14%, just
because we don't waste time on reallocations during append().

Before:
   10000            439754 ns/op          218859 B/op        428 allocs/op
ok      github.com/CityOfZion/neo-go/integration        5.423s

After:
   10000            369833 ns/op           87209 B/op        412 allocs/op
ok      github.com/CityOfZion/neo-go/integration        4.612s
2019-12-13 19:57:17 +03:00
Roman Khimov
c9257c3de4 vm: optimize Next() in Context
Creating a new BinReader for every instruction is a bit too much and it adds
about 1% overhead on block import (and actually is quite visible in the VM
profiling statistics). So use a bit more ugly but efficient method.
2019-12-13 19:57:17 +03:00
Roman Khimov
a9401e2ec7 core: do not write new dao elements into DB
It's useless work being done before it's actually needed. These (updated with
new values) are going to be written with some kind of Put anyway, so writing
them here is just a waste of time.
2019-12-13 19:57:17 +03:00
Roman Khimov
6896b40dee state: use open-coded array (de)serialization for accounts
We're spending a lot of time here, 100K blocks import starting at 1.4M, before
this patch:
real    4m17,748s
user    6m23,316s
sys     0m37,866s

After:
real    3m54,968s
user    5m56,547s
sys     0m39,398s

9% is quite a substantial improvement to justify this change.
2019-12-13 19:57:17 +03:00
Roman Khimov
0d0a27d271 core: add cachedDao to cache accounts and contracts
Importing 100K blocks starting at 1.4M, before this patch:
real    6m0,356s
user    8m52,293s
sys     0m47,372s

After this patch:
real    4m17,748s
user    6m23,316s
sys     0m37,866s

Almost 30% better.
2019-12-13 19:57:17 +03:00
Roman Khimov
212cf44e26 core: get data from dao in interops
It's more logically correct as Blockchain can be several transactions behind
in terms of its state.
2019-12-13 17:17:14 +03:00
Roman Khimov
0abd55c2c2 vm: add cached ScriptHash() to Context
Avoid recalculating it over and over again in interop.
2019-12-13 17:05:03 +03:00
Roman Khimov
0afaff2f79 vm: microoptimize RegisterInteropFuncs()
Avoid useless copying.
2019-12-13 17:02:28 +03:00
Evgenii Stratonikov
0efd9a6062 consensus: fix payload sign test
Do not fill verification script randomly as there is a probability
for it to be executed sucessfully.

time="2019-12-12T17:24:22+03:00" level=info msg="blockchain persist completed" blockHeight=0 headerHeight=0 persistedBlocks=0 persistedKeys=15 took="54.474µs"
time="2019-12-12T17:24:23+03:00" level=info msg="blockchain persist completed" blockHeight=0 headerHeight=0 persistedBlocks=0 persistedKeys=15 took="49.312µs"
2019-12-12T17:24:24.026+0300    DEBUG   can't verify payload from #%d1  {"module": "dbft"}
--- FAIL: TestPayload_Sign (0.00s)
    payload_test.go:302:
                Error Trace:    payload_test.go:302
                Error:          Should be false
                Test:           TestPayload_Sign
FAIL
coverage: 75.8% of statements
FAIL    github.com/CityOfZion/neo-go/pkg/consensus      2.145s
2019-12-13 12:09:51 +03:00
Evgenii Stratonikov
8a2e1c3d0a io: remove ReadLE/BE and WriteLE/BE
It was replaced with faster specialized versions earlier.
2019-12-13 11:38:28 +03:00
Roman Khimov
09223236f9 core: add Persist() to dao()
Hide its internals a little, which is gonna be useful for future composition.
2019-12-12 21:17:13 +03:00
Roman Khimov
eb1749d190 core: remove persisting from GetValidators()
It's a getter function and even though it's quite fancy with its transactions
processing (for consensus operation) it shouldn't ever change the state of the
Blockchain. If we're to change anything here these changes may conflict with
the actual block processing later or may lead to broken state (if transactions
won't be approved for some reason).
2019-12-12 21:14:33 +03:00
Roman Khimov
31dbc35279 core: gofmt dao_test.go 2019-12-12 21:05:36 +03:00
Roman Khimov
c5ed3b788b core: simplify dao creation with newDao() 2019-12-12 21:04:55 +03:00
Roman Khimov
8b3080b972 io: rename Read/WriteBytes to Read/WriteB
go vet is not happy about them:
  pkg/io/binaryReader.go:92:21: method ReadByte() byte should have signature ReadByte() (byte, error)
  pkg/io/binaryWriter.go:75:21: method WriteByte(u8 byte) should have signature WriteByte(byte) error
2019-12-12 20:19:50 +03:00
Roman Khimov
0b14916d79 io: use optimized Read/WriteUXX for Read/WriteVarUint() 2019-12-12 20:19:50 +03:00
Roman Khimov
54d888ba70 io: add type-specific read/write methods
This seriously improves the serialization/deserialization performance for
several reasons:
 * no time spent in `binary` reflection
 * no memory allocations being made on every read/write
 * uses fast ReadBytes everywhere it's appropriate

It also makes Fixed8 Serializable just for convenience.
2019-12-12 20:19:50 +03:00
Roman Khimov
89d7f6d26e core/tx: microoptimize block/tx hashing
Don't hash the data twice.
2019-12-12 18:01:30 +03:00
Roman Khimov
7e83078d13 hash: remove useless no-op decoding
It changes nothing here.
2019-12-12 17:58:34 +03:00
Vsevolod Brekelov
c93a8d2bc4 core: extracted same logic to separate methods 2019-12-11 13:14:45 +03:00
Vsevolod Brekelov
c1f39d5c7b internal: moved testutil method to internal package 2019-12-11 13:14:43 +03:00