This error message makes no sense when shutting down the server:
2020-06-25T19:29:53.251+0300 ERROR failed to start RPC server {"error": "http: Server closed"}
And ListenAndServer is documented to always return non-nil error one of which
is http.ErrServerClosed. This should also fix the following test failure:
==================
WARNING: DATA RACE
Read at 0x00c000254243 by goroutine 49:
testing.(*common).logDepth()
/usr/local/go/src/testing/testing.go:665 +0xa1
testing.(*common).Logf()
/usr/local/go/src/testing/testing.go:658 +0x8f
testing.(*T).Logf()
<autogenerated>:1 +0x75
go.uber.org/zap/zaptest.testingWriter.Write()
/go/pkg/mod/go.uber.org/zap@v1.10.0/zaptest/logger.go:130 +0x11f
go.uber.org/zap/zaptest.(*testingWriter).Write()
<autogenerated>:1 +0xa9
go.uber.org/zap/zapcore.(*ioCore).Write()
/go/pkg/mod/go.uber.org/zap@v1.10.0/zapcore/core.go:90 +0x1c3
go.uber.org/zap/zapcore.(*CheckedEntry).Write()
/go/pkg/mod/go.uber.org/zap@v1.10.0/zapcore/entry.go:215 +0x1e7
go.uber.org/zap.(*Logger).Error()
/go/pkg/mod/go.uber.org/zap@v1.10.0/logger.go:203 +0x95
github.com/nspcc-dev/neo-go/pkg/rpc/server.(*Server).Start()
/go/src/github.com/nspcc-dev/neo-go/pkg/rpc/server/server.go:179 +0x5c5
Previous write at 0x00c000254243 by goroutine 44:
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:900 +0x353
testing.tRunner()
/usr/local/go/src/testing/testing.go:913 +0x1bb
Goroutine 49 (running) created at:
github.com/nspcc-dev/neo-go/pkg/rpc/server.initClearServerWithInMemoryChain()
/go/src/github.com/nspcc-dev/neo-go/pkg/rpc/server/server_helper_test.go:69 +0x305
github.com/nspcc-dev/neo-go/pkg/rpc/server.initServerWithInMemoryChain()
/go/src/github.com/nspcc-dev/neo-go/pkg/rpc/server/server_helper_test.go:78 +0x3c
github.com/nspcc-dev/neo-go/pkg/rpc/server.testRPCProtocol()
/go/src/github.com/nspcc-dev/neo-go/pkg/rpc/server/server_test.go:805 +0x53
github.com/nspcc-dev/neo-go/pkg/rpc/server.TestRPC.func1()
/go/src/github.com/nspcc-dev/neo-go/pkg/rpc/server/server_test.go:793 +0x44
testing.tRunner()
/usr/local/go/src/testing/testing.go:909 +0x199
Goroutine 44 (finished) created at:
testing.(*T).Run()
/usr/local/go/src/testing/testing.go:960 +0x651
github.com/nspcc-dev/neo-go/pkg/rpc/server.TestRPC()
/go/src/github.com/nspcc-dev/neo-go/pkg/rpc/server/server_test.go:792 +0x5d
testing.tRunner()
/usr/local/go/src/testing/testing.go:909 +0x199
==================
ValidatorsCount is not initialized at block 0 with C# node (the first voter
initializes it) and until that initialization happens the standby validators
list is being returned as is without sorting.
Fixes state mismatch for the key ffffffff0e00000000000000000000000000000001 in
the first blocks.
It also affects tests as now the first validator is different and it receives
the network fees.
There is no such thing as high/low priority transactions, as there are
no free transactions anymore and they are ordered by fees contained
in transaction itself.
Closes#1063.
It's just JSON, io.Serializable is only used for DB storage where the length
should be obtained from the stream. Fixes:
2020-06-18T22:14:10.571+0300 WARN contract invocation failed {"tx": "1ffd475a9c246495d6206cb80a9a78e9d14a433ded60cd37aa87d897655606e1", "block": 25893, "error": "error encountered at instruction 3696 (SYSCALL): failed to invoke syscall: invalid character ':' after top-level value"}
We make it explicit in the appropriate Block/Transaction structures, not via a
singleton as C# node does. I think this approach has a bit more potential and
allows better packages reuse for different purposes.
It doesn't affect anything yet, but it's going to be used in the future for
network-specific behavior. It also renames short '--timeout' form to '-s'
avoiding conlict with '-t' used for '--testnet'.
1. Remove GetScript, IsPayable, GetStorageContext.
2. Revert 82319538 related to GetStorageContext.
3. Rename Migrate to Update.
4. Move remaining to System.Contract.*.
Related #1031.
Two changes being done here, because they require a lot of updates to
tests. Now we're back into version 0 and we only have one type of
transaction.
It also removes GetType and GetScript interops, both are obsolete in Neo 3.
Our block.Block was JSONized in a bit different fashion than result.Block in
its NextConsensus and Index fields. It's not good for notifications because
third-party clients would probably expect to see the same format. Also, using
completely different Block representation is probably making our client a bit
weaker as this representation is harder to use with other neo-go components.
So use the same approach we took for Transactions and wrap block.Block which is
to be serialized in proper way.
Fix `Script` JSONization along the way, 3.0 node wraps it within `witnesses`.
Note that the protocol differs a bit from #895 in its notifications format,
to avoid additional server-side processing we're omitting some metadata like:
* block size and confirmations
* transaction fees, confirmations, block hash and timestamp
* application execution doesn't have ScriptHash populated
Some block fields may also differ in encoding compared to `getblock` results
(like nonce field).
I think these differences are unnoticieable for most use cases, so we can
leave them as is, but it can be changed in the future.
It will be important for proper subscription testing and it doesn't hurt even
though technically we've got two http servers listening after this change (one
is a regular Server's http.Server and one is httptest's Server). Reusing
rpc.Server would be nice, but it requires some changes to Start sequence to
start Listener with net.Listen and then communicate back its resulting
Addr. It's not very convenient especially given that no other code needs it,
so doing these changes just for a bit cleaner testing seems like and
overkill.
Update config appropriately. Update Start comment along the way.
1. Slot is a new mechanism for storing variables during execution
which is more convenient than alt.stack. This commit implements
support for slot opcodes in both vm and compiler.
2. Remove old alt.stack opcodes.
3. Do not process globals at the start of every function, but instead
load them single time at main.
Rename THROWIFNOT to ASSERT, add ABORT opcode.
ABORT cannot be caught, but the implementation should be postponed until
exception handling is implemented.
The original intention here was to check for server reaction in presence of
validation errors (response.ErrValidationFailed) and it was very easy to make
validation fail for block without transactions in Neo 2. But in Neo 3
transactionless blocks are perfectly valid (see
29d321b5e1) which broke this test even though we
didn't see it until websocket addition (which required
8cec6694ae to make this test work initially).
So make a valid block and then spoil its verification script.
1. Dropped `Base.ConsensusData` block field
2. Added `Block.ConsensusData` field with `Nonce` and `PrimaryIndex`
3. Removed "Neo.Header.GetConsensusData" and
"AntShares.Header.GetConsensusData" interops
Our test chain is used in core,rpc and (in future) interop packages.
It is better to have all related declarations in one place to avoid
code duplication.
In NEO3 SYSCALL opcode has 4-byte ID parameter.
This commit removes support for string-based syscalls and
changes SYSCALL's parameter to be fixed 4-byte value.
1. closes#841
2. Commented out test cases where binary transaction are used.
These test cases marked with `TODO NEO3.0: Update binary` and need to be
updated.
3. Updated other tests.
4. Added cache to calculateValidUntilBlock() RPC-client method.
1. Closes#840: added Nonce field to transaction.Transaction and
removed Nonce field from transaction.MinerTx
2. Added following methods to different tx types:
- NewMinerTx()
- NewMinerTxWithNonce(...)
- NewEnrollmentTx(...)
- NewIssueTx()
- NewPublishTx(...)
- NewRegisterTx(...)
- NewStateTx(...)
in order to avoid code duplication when new transaction is created.
3. Commented out test cases where binary transaction/block are used.
These test cases marked with `TODO NEO3.0: Update binary` and need to be
updated.
4. Updated other tests
5. Added constant Nonce to GoveringTockenTx, UtilityTokenTx and genesis
block to avoid data variability. Also marked with TODO.
Modified result.AssetState:
- removed `FeeMode` and `FeeAddress` fields
- fixed json name of `ID` and `AssetType` fields
to be consistent with C# RPC server
Passing WIF directly in the command line is not something we should be doing.
Also split netfee and sysfee in the RPC as they're different (and add a script
attribute for free transactions).
That's how it was intended to behave originally. One thing questionable here
is contract price (policy thing, basically) being moved to smartcontract
package, but it's probably fine for NEO 2.0 (as it won't change) and we'll
make something better for NEO 3.0.
They have it specified right in the transaction. Unfortunately, this little
change rendered invalid our RPC test chain, but I think it became even better
after it, especially given that chain generation is a nice test by itself, so
it should be running as a regular test.
And drop doc.go from server package as it duplicates docs/rpc.md and has no
value of its own (TODO list is managed with GitHub issues, really). Also, RPC
server is not really expected to be used by non-neo-go packages (contrary to
the client).
Merged two types:
- smartcontract.ParamType
- rpc.StackParamType
into single one:
- smartcontract.ParamType
as they duplicated the functionality.
NOTE: type smartcontract.MapType was added (as in C# implementation).
From now, list of supported smartcontract parameter types:
UnknownType
SignatureType
BoolType
IntegerType
Hash160Type
Hash256Type
ByteArrayType
PublicKeyType
StringType
ArrayType
MapType
InteropInterfaceType
VoidType
sendrawtransaction just returns a bool, sendtoaddress returns a proper
transaction and that should be the same as the one we have in
TransactionOutputRaw.
Mostly as is, no real effort done yet to optimize them, so there are still a
lot of duplicates there, but at least we sort them out into different smaller
packages.
Both are very useful outside of the core, this change also makes respective
transactions initialize with the package as they don't depend on any kind of
input and it makes no sense recreating them again and again on every use.
Changed returncode of getrowtransaction method in case when transaction
with specified hash does not exists. Now it returns error with code -100
instead of -32602 (as in c# node)
Returning error string as a result (not an error) is utterly wrong, but C#
implementation just returns a zero balance for unknown addresses, so we should
follow that.
If blockchain is not closed, logging in defer can occur
after test has finished, which will lead to a panic with
"Log in goroutine after Test* has completed".
This seriously improves the serialization/deserialization performance for
several reasons:
* no time spent in `binary` reflection
* no memory allocations being made on every read/write
* uses fast ReadBytes everywhere it's appropriate
It also makes Fixed8 Serializable just for convenience.
add dao which takes care about all CRUD operations on storage
remove blockchain state since everything is stored on change
remove storage operations from structs(entities)
move structs to entities package
It reduces heap pressure a little for these elements as we don't have to
allocate/free them individually. And they're directly tied to transactions or
block, not being shared or anything like that, so it makes little sense for
them to be pointer-based. It only makes building transactions a little easier,
but that's obviously a minor usecase.
Drop wif.GetVerificationScript(), drop
smartcontract.CreateSignatureRedeemScript(), add GetVerificationScript()
directly to the PublicKey and use it everywhere.
This allows easier reuse of opcodes and in some cases allows to eliminate
dependencies on the whole vm package, like in compiler that only needs opcodes
and doesn't care about VM for any other purpose.
And yes, they're opcodes because an instruction is a whole thing with
operands, that's what context.Next() returns.
Param getters were redone to return errors because otherwise bad FuncParam
values could lead to panic. FuncParam itself might be not the most elegant
solution, but it works good enough for now.
This PR does 3 things:
adds array parameter unmarshalling
extend Param with convenient methods
refactor tests into using tables to make it easier add new tests
(part of #347 solution)
There is a difference in interpretation of what a block count is. neo-go nodes
currently respond to this request with the latest block number which is the
same number that neoscan.io shows. However, C# nodes deliberately do add one
to this number when answering to the getblockcount request to account for the
genesis block number 0.
This patch makes us consistent with C# nodes wrt to getblockcount behaviour.
This one enables our RPC to be called from the browser if there is a
need. It's insecure and not standards-compliant, thus this behaviour is
configurable is not enabled by default. It makes our node with this workaround
enabled compatible with neo-mon monitoring.
Originally debugged by @anatoly-bogatyrev in #464.
Extend Blockchainer with one more method to spawn a VM for test runs and use
it to run scripts. Gas consumption is not counted or limited in any way at the
moment (see #424).
This adds the following verifications:
* merkleroot check
* index check
* timestamp check
* witnesses verification
VerifyWitnesses is also renamed to verifyTxWitnesses here to not confuse it
with verifyBlockWitnesse and to hide it from external access (no users at the
moment).
We want to get a full block, so it has to have transactions
inside. Unfortunately our tests were used to this wrong behavior and utilized
completely bogus transactions without data that couldn't be persisted, so fix
that also.
Commit 578ac414d4 was wrong in that it saved
only a part of the block, so depending on how you use blockchain, you may
still see that the block was not really processed properly. To really fix it
this commit introduces intermediate storage layer in form of memStore, which
actually is a MemoryStore that supports full Store API (thus easily fitting
into the existing code) and one extension that allows it to flush its data to
some other Store.
It also changes AddBlock() semantics in that it only accepts now successive
blocks, but when it does it guarantees that they're properly added into the
Blockchain and can be referred to in any way. Pending block queing is now
moved into the server (see 8c0c055ac657813fe3ed10257bce199e9527d5ed).
So the only thing done with persist() now is just a move from memStore to
Store which probably should've always been the case (notice also that
previously headers and some other metadata was written into the Store
bypassing caching/batching mechanism thus leading to some inefficiency).
This changes the Blockchain to also return unpersisted (theoretically, verified
in the AddBlock!) blocks and transactions, making Add/Get interfaces
symmetrical. It allows to turn Persist into internal method again and makes it
possible to enable transaction check in GetBlock(), thus fixing #366.
earlier we had an issue with failing test in #353 and other one #305.
Reworked these test to have in-memory database. This led to multiple
changes: made some functions like Hash and Persist public(otherwise
it's not possible to control state of the blockchain); removed
unit_tests storage package which was used mainly for leveldb in unit
tests.
I see these tests not really good since they look like e2e tests and
as for me should be run in separate step against dockerized env or
in case we want to check rpc handler we might want to rework it in order
to have interface for proper unit tests.
As for me this patchset at least makes as safe with not removing totally
previous tests and at the same time CircleCI will be happy now.
The logic here is that we'll have all binary encoding/decoding done via our io
package, which simplifies error handling. This functionality doesn't belong to
util, so it's moved.
This also expands BufBinWriter with Reset() method to fit the needs of core
package.
add close function to storage interface
add common defer function call which will close db connection
remove context as soon as it's not needed anymore
updated unit tests
PublicKey() for PrivateKey now just can't fail and it makes no sense to return
an error from it. There is a lot of associated functionality for which this
also is true, so adjust it accordingly and simplify a lot of code.
Golint:
pkg/rpc/rpc.go:15:67: exported method GetBlock returns unexported type *rpc.response, which can be annoying to use
pkg/rpc/rpc.go:82:64: exported method GetRawTransaction returns unexported type *rpc.response, which can be annoying to use
pkg/rpc/rpc.go:97:52: exported method SendRawTransaction returns unexported type *rpc.response, which can be annoying to use
Refs. #213.
pkg/rpc/neoScanBalanceGetter.go:54:56: method parameter assetIdUint should be assetIDUint
pkg/rpc/neoScanBalanceGetter.go:62:3: var assetId should be assetID
pkg/rpc/server_test.go:27:5: var testRpcCases should be testRPCCases
pkg/rpc/txTypes.go:19:3: struct field assetId should be assetID
pkg/rpc/txTypes.go:39:35: interface method parameter assetId should be assetID
pkg/rpc/types.go:115:2: struct field TxId should be TxID
Refs. #213.
Fixes things like:
* exported type/method/function X should have comment or be unexported
* comment on exported type/method/function X should be of the form "X ..."
(with optional leading article)
Refs. #213.
And drop associated _pkg.dev remnants (refs. #307).
Original `dev` branch had two separate packages for public and private keys,
but those are so intertwined (`TestHelper` subpackage is a proof) that it's
better unite them and all associated code (like WIF and NEP-2) in one
package. This patch also:
* creates internal `keytestcases` package to share things with wallet (maybe
it'll be changed in some future)
* ports some tests from `dev`
* ports Verify() method for public key from `dev`
* expands TestPrivateKey() with public key check
* Added new config attributes: 'SecondsPerBlock','LowPriorityThreshold'
* Added new files:
* Added new method: CompareTo
* Fixed empty Slice case
* Added new methods: LessThan, GreaterThan, Equal, CompareTo
* Added new method: InputIntersection
* Added MaxTransactionSize, GroupOutputByAssetID
* Added ned method: ScriptHash
* Added new method: IsDoubleSpend
* Refactor blockchainer, Added Feer interface, Verify and GetMemPool method
* 1) Added MemPool
2) Added new methods to satisfy the blockchainer interface: IsLowPriority, Verify, GetMemPool
* Added new methods: RelayTxn, RelayDirectly
* Fixed tests
* Implemented RPC server method sendrawtransaction
* Refactor getrawtransaction, sendrawtransaction in separate methods
* Moved 'secondsPerBlock' to config file
* Implemented Kim suggestions:
1) Fixed data race issues
2) refactor Verify method
3) Get rid of unused InputIntersection method due to refactoring Verify method
4) Fixed bug in https://github.com/CityOfZion/neo-go/pull/174#discussion_r264108135
5) minor simplications of the code
* Fixed minor issues related to
1) space
2) getter methods do not need pointer on the receiver
3) error message
4) refactoring CompareTo method in uint256.go
* Fixed small issues
* Use sync.RWMutex instead of sync.Mutex
* Refined (R)Lock/(R)Unlock
* return error instead of bool in Verify methods
* Added utility function GetVarSize
* 1) Added Size method: this implied that Fixed8 implements now the serializable interface. 2) Added few arithmetic operation (Add, Sub, div): this will be used to calculated networkfeeand feePerByte. Changed return value of the Value() method to int instead of int64. Modified fixed8_test accordingly.
* Implemented Size or MarshalJSON method.
- Structs accepting the Size method implement the serializable interface.
- Structs accepting the MarshalJSON method implements the customized json marshaller interface.
* Added fee calculation
* Implemented rcp server method GetRawTransaction
* Updated Tests
* Fixed:
1) NewFixed8 will accept as input int64
2) race condition affecting configDeafault, blockchainDefault
* Simplified Size calculation
* 1) Removed global variable blockchainDefault, configDefault
2) Extended Blockchainer interface to include the methods: References, FeePerByte, SystemFee, NetworkFee
3) Deleted fee_test.go, fee.go. Moved corresponding methods to blockchain_test.go and blockchain.go respectively
4) Amended tx_raw_output.go
* Simplified GetVarSize Method
* Replaced ValueAtAndType with ValueWithType
* Cosmetic changes + Added test case getrawtransaction_7
* Clean up Print statement
* Filled up keys
* Aligned verbose logic with the C#-neo implementation
* Implemented @Kim requests
Refactor server_test.go
* Small fixes
* Fixed verbose logic
Added more tests
Cosmetic changes
* Replaced assert.NoError with require.NoError
* Fixed tests by adding context.Background() as argument
* Fixed tests
* Fix#130: Wrong answer for RPC server method getaccountstate
- fixed RPC Server response
- fixed RPC Server tests
- remove unused package from go.mod
* add index and type checker
* fix review comments (thx @aprasolova)
- balancer / wif / http.Client could be with data race
- add getter / setter with sync.Mutex
- now http.Client is pointer
- now you can provide your http.Client to rpcClient
* Implemented rcp method GetAccountState
* code clean up
* Removed empty line
* Used consistently github.com/pkg/errors package. Amended error message
* Get rid of fmt.Sprintf and use either errors.Errorf or errors.Wrapf
* cosmetic changes
* Fixed small incosistencies related to comments and variable name
* For testing purposes we have to move the method newBlockchain from the cli/server/server.go to pkg/core/blockchain.go. In addition we have to rename it to NewBlockchainLevelDB in order to be able to export it and avoid naming conflicts. In future we still need to think how to switch between different blockchain implementation easily but for the time being this is not possible.
* Added unit tests for the rpc server
* Added unit_testnet chain fixture
* fixed port number
* Added errors handling
* move unit_testnet chain from 'cli/chains' to 'pkg/rpc/chains'
* func to get private key from raw bytes
* added function to create raw transfer tx
* fixes
* more fixes
* prettify code and comments; neoscan interaction put in dedicated files
* Few tweaks to improve output of `getblock`
* Adds few more fields and corrects witness
* Bumps version
* Only reverse when marshalling for moment
* Adds README for rpc package
* Few updates
* Typo
* Adds link in main readme
* Fix readme link
* implemented operation and param flags in the cli invoke cmd.
* reverted prev changes and added debug flag for compiling.
* change transactionType variable to Type, for package convention
* index support for arrays.
* implemented builtin (len) for the compiler.
* bumped version -> 0.35.0
* updated compiler README and changed invoke to testinvoke.
* Adds basic RPC supporting files
* Adds interrupt handling and error chan
* Add getblock RPC method
* Update request structure
* Update names of nodes
* Allow bad addresses to be registered in discovery externally
* Small tidy up
* Few tweaks
* Check if error is close error in tcp transport
* Fix tests
* Fix priv port
* Small tweak to param name
* Comment fix
* Remove version from server
* Moves submitblock to TODO block
* Remove old field
* Bumps version and fix hex issues
* Started RPC package to allow querying balances and sending raw transactions for sc's
* integrate invoke cmd in cli
* bumped version
* added sendrawtransaction to the rpc client.
