Commit graph

2039 commits

Author SHA1 Message Date
Roman Khimov
66ddeccdad
Merge pull request #2813 from nspcc-dev/fix-state-reset
core: fix broken state reset
2022-11-23 13:43:42 +07:00
Anna Shaleva
f3ef2890f0 core: check headers at the proper state on state reset
And fix the comment along the way.
2022-11-23 09:16:33 +03:00
Anna Shaleva
b27a9bcf95 core: adjust info message for proper-stated chains
Make it prettier for those cases when `db reset` command was called
after interrupted reset.
2022-11-22 11:53:39 +03:00
Anna Shaleva
b82374823e core: increase persist batch size for reset storage changes 2022-11-22 11:53:39 +03:00
Anna Shaleva
bdc42cd595 core: reset blocks, txs and AERs in several stages
Sometimes it can be hard to persist all changes at ones, the process
can take almost all RAM and a lot of time. Here's the example of reset
for mainnet from 2.4M to 1:
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -m --height 1
2022-11-20T17:16:48.236+0300	INFO	MaxBlockSize is not set or wrong, setting default value	{"MaxBlockSize": 262144}
2022-11-20T17:16:48.236+0300	INFO	MaxBlockSystemFee is not set or wrong, setting default value	{"MaxBlockSystemFee": 900000000000}
2022-11-20T17:16:48.237+0300	INFO	MaxTransactionsPerBlock is not set or wrong, using default value	{"MaxTransactionsPerBlock": 512}
2022-11-20T17:16:48.237+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T17:16:48.240+0300	INFO	restoring blockchain	{"version": "0.2.6"}
2022-11-20T17:16:48.297+0300	INFO	initialize state reset	{"target height": 1}
2022-11-20T17:16:48.300+0300	INFO	trying to reset blocks, transactions and AERs
2022-11-20T17:19:29.313+0300	INFO	blocks, transactions ans AERs are reset	{"took": "2m41.015126493s", "keys": 3958420}
...
```
To avoid OOM killer, split blocks reset into multiple stages. It increases
operation time due to intermediate DB persists, but makes things cleaner, the
result for almost the same DB height with the new approach:
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -m --height 1
2022-11-20T17:39:42.023+0300	INFO	MaxBlockSize is not set or wrong, setting default value	{"MaxBlockSize": 262144}
2022-11-20T17:39:42.023+0300	INFO	MaxBlockSystemFee is not set or wrong, setting default value	{"MaxBlockSystemFee": 900000000000}
2022-11-20T17:39:42.023+0300	INFO	MaxTransactionsPerBlock is not set or wrong, using default value	{"MaxTransactionsPerBlock": 512}
2022-11-20T17:39:42.023+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T17:39:42.026+0300	INFO	restoring blockchain	{"version": "0.2.6"}
2022-11-20T17:39:42.071+0300	INFO	initialize state reset	{"target height": 1}
2022-11-20T17:39:42.073+0300	INFO	trying to reset blocks, transactions and AERs
2022-11-20T17:40:11.735+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 1, "took": "29.66363737s", "keys": 210973}
2022-11-20T17:40:33.574+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 2, "took": "21.839208683s", "keys": 241203}
2022-11-20T17:41:29.325+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 3, "took": "55.750698386s", "keys": 250593}
2022-11-20T17:42:12.532+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 4, "took": "43.205892757s", "keys": 321896}
2022-11-20T17:43:07.978+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 5, "took": "55.445398156s", "keys": 334822}
2022-11-20T17:43:35.603+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 6, "took": "27.625292032s", "keys": 317131}
2022-11-20T17:43:51.747+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 7, "took": "16.144359017s", "keys": 355832}
2022-11-20T17:44:05.176+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 8, "took": "13.428733899s", "keys": 357690}
2022-11-20T17:44:32.895+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 9, "took": "27.718548783s", "keys": 393356}
2022-11-20T17:44:51.814+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 10, "took": "18.917954658s", "keys": 366492}
2022-11-20T17:45:07.208+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 11, "took": "15.392642196s", "keys": 326030}
2022-11-20T17:45:18.776+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 12, "took": "11.568255716s", "keys": 299884}
2022-11-20T17:45:25.862+0300	INFO	last batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 13, "took": "7.086079594s", "keys": 190399}
2022-11-20T17:45:25.862+0300	INFO	blocks, transactions ans AERs are reset	{"took": "5m43.791214084s", "overall persisted keys": 3966301}
...
```
2022-11-22 11:53:39 +03:00
Anna Shaleva
d67f0df516 core: reset block headers together with header height info
We need to keep the headers information consistent with header batches
and headers. This comit fixes the bug with failing blockchain
initialization on recovering from state reset interrupted after the
second stage (blocks/txs/AERs removal):
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -t --height 83000
2022-11-20T16:28:29.437+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T16:28:29.440+0300	INFO	restoring blockchain	{"version": "0.2.6"}
failed to create Blockchain instance: could not initialize blockchain: could not get header 1898cd356a4a2688ed1c6c7ba1fd6ba7d516959d8add3f8dd26232474d4539bd: key not found
```
2022-11-22 11:53:39 +03:00
Anna Shaleva
283da8f599 core: use DAO-provided block height during during state reset
Don't use cache because it's not yet initialized. Also, perform
safety checks only if state reset wasn't yet started. These fixes
alloww to solve the following problem while recovering from
interrupted state reset:
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -t --height 83000
2022-11-20T15:51:31.431+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T15:51:31.434+0300	INFO	restoring blockchain	{"version": "0.2.6"}
failed to create Blockchain instance: could not initialize blockchain: current block height is 0, can't reset state to height 83000
```
2022-11-22 11:53:39 +03:00
Anna Shaleva
7d55bf2cc1 core: log persisted storage item batches count during state reset 2022-11-22 11:53:39 +03:00
Anna Shaleva
f52451e582 core: fix state reset with broken contract
Sync up with #2802, bad contract -> no contract ID at all.
2022-11-22 11:53:39 +03:00
Anna Shaleva
ecda07736e core: stop storage items reset after any seek error 2022-11-22 11:53:39 +03:00
Anna Shaleva
bfe7aeae7b core: stop storage items reset after the first persist error
It's a bug, we mustn't continue if something bad had happend on persist,
otherwise this error will be overwritten by subsequent successfull persist.
2022-11-22 11:53:39 +03:00
Anna Shaleva
235518eb6c core: reset batch counter to zero after each persist in resetStateInternal
It's a bug, otherwise we'll persist each storage item after 10K-th one,
that's the reason of abnormous long storage items resetting stage.
2022-11-22 11:53:39 +03:00
Anna Shaleva
9f23fafc03 core: improve logging of resetStateInternal
Inform when starting subsequent stage, inform about keys persisted.
2022-11-22 11:53:36 +03:00
Roman Khimov
cb64957af5 vm: don't use Stack for istack
We don't use all of the Stack functionality for it, so drop useless methods
and avoid some interface conversions. It increases single-node TPS by about
0.9%, so nothing really important there, but not a bad change either. Maybe it
can be reworked again with generics though.
2022-11-18 11:35:29 +03:00
Roman Khimov
ab0ff63ce1
Merge pull request #2804 from nspcc-dev/check-aer-sub
rpc: fix subscribers locking logic and properly drain poll-based waiter receiver
2022-11-17 04:24:35 +07:00
Anna Shaleva
6dbae7edc4 rpcclient: fix WS-client unsubscription process
Do not block subscribers until the unsubscription request to RPC server
is completed. Otherwise, another notification may be received from the
RPC server which will block the unsubscription process.

At the same time, fix event-based waiter. We must not block the receiver
channel during unsubscription because there's a chance that subsequent
event will be sent by the server. We need to read this event in order not
to block the WSClient's readloop.
2022-11-16 23:44:30 +03:00
Roman Khimov
822722bd2e native: ignore decoding errors during cache init
Bad contract -> no contract. Unfortunately we've got a broken
6f1837723768f27a6f6a14452977e3e0e264f2cc contract on the mainnet which can't
be decoded (even though it had been saved successfully), so this is a
temporary fix for #2801 to be able to start mainnet node after shutdown.
2022-11-16 12:00:28 +03:00
Roman Khimov
f78231fd9c
Merge pull request #2773 from nspcc-dev/state-reset
core: implement state reset
2022-11-10 22:26:43 +07:00
Anna Shaleva
e7effef052 core: reuse appendTokenTransferInfo from appendTokenTransfer 2022-11-10 18:08:18 +03:00
Anna Shaleva
bd6bb9e9e2 core: allow to reset blockchain state 2022-11-10 18:08:17 +03:00
Anna Shaleva
ec9317d5b4 core: fix typo 2022-11-10 17:58:06 +03:00
Anna Shaleva
1dac45bbbb core: add ability to check whether blockchain is running 2022-11-10 16:47:04 +03:00
Roman Khimov
8746d9877e keys: move from btcd/btcec to decred/secp256k1 package
There is a security issue found in github.com/btcsuite/btcd that we don't care
about (we're only using 256k1 implementation), but GitHub complains about
it. We could update to github.com/btcsuite/btcd/btcec/v2, but it's now just a
thin wrapper over github.com/decred/dcrd/dcrec/secp256k1/v4, so we better use
it directly.
2022-11-08 17:59:59 +03:00
Roman Khimov
0c3b03617e core: optimize HasBlock check for recent blocks
When block is being spread through the network we can get a lot of invs with
the same hash. Some more stale nodes may also announce previous or some
earlier block. We can avoid full DB lookup for them and minimize inv handling
time (timeouts in inv handler had happened in #2744).

It doesn't affect tests, just makes node a little less likely to spend some
considerable amount of time in the inv handler.
2022-10-21 21:21:58 +03:00
Anna Shaleva
63fddb3f1a core: close BoltDB on failed root bucket creation 2022-10-10 10:12:34 +03:00
Anna Shaleva
95cbddf19e cli: use custom logger to filter out runtime.Log messages
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go vm -p

    _   ____________        __________      _    ____  ___
   / | / / ____/ __ \      / ____/ __ \    | |  / /  |/  /
  /  |/ / __/ / / / /_____/ / __/ / / /____| | / / /|_/ /
 / /|  / /___/ /_/ /_____/ /_/ / /_/ /_____/ |/ / /  / /
/_/ |_/_____/\____/      \____/\____/      |___/_/  /_/

NEO-GO-VM > loadgo ./1-print/1-print.go
READY: loaded 21 instructions
NEO-GO-VM 0 > run
2022-10-07T15:28:20.461+0300	INFO	runtime log	{"tx": "", "script": "db03ceb3f672ee8cd0d714989b4d103ff7eed2f3", "msg": "Hello, world!"}
[]
```
2022-10-07 15:57:33 +03:00
Anna Shaleva
0db4e8d62c core: allow to perform storage search within given amount of DAO layers 2022-10-07 15:56:34 +03:00
Anna Shaleva
79e13f73d8 core, rpc: move getFakeNextBlock to Blockchain
It's needed for VM CLI as far and may be improved later.
2022-10-07 15:56:34 +03:00
Anna Shaleva
a91cf2a007 core: set default SecondsPerBlock value on blockchain creation
As mentioned in the node configuration docs.
2022-10-07 15:56:34 +03:00
Anna Shaleva
70e59d83c9 docs: fix supported database types 2022-10-07 15:56:34 +03:00
Anna Shaleva
2f5137e9b7 core: allow RO mode for Bolt and Level 2022-10-07 15:56:29 +03:00
Anna Shaleva
cbdd45cc96 core: return error on root BoltDB bucket creation if so 2022-10-06 14:01:56 +03:00
Anna Shaleva
03a1cf9f59 core: simplify newLevelDBForTesting function 2022-10-06 14:01:56 +03:00
Anna Shaleva
78cd2b4566 core: add example to VUB comment 2022-10-06 10:32:18 +03:00
Roman Khimov
317dd42513 *: use uint*Size and SignatureLen constants where appropriate 2022-10-05 10:45:52 +03:00
Roman Khimov
79887f9d78 runtime: check notifications against ABI
Related to #2703, just a logged thing for now.
2022-10-04 17:52:38 +03:00
Roman Khimov
1b753cd4bc native: add some tests for stdlib's atoi
See neo-project/neo#2804 and neo-project/neo#2813. We're already compatible.
2022-09-19 16:18:53 +03:00
Roman Khimov
97193cf337 golangci: add predeclared linter
These can be confusing.
2022-09-02 18:36:26 +03:00
Roman Khimov
63f212f4b3 golangci: enable/fix misspell 2022-09-02 18:36:26 +03:00
Roman Khimov
4f3ffe7290 golangci: enable errorlint and fix everything it found 2022-09-02 18:36:23 +03:00
Roman Khimov
e569edc841 wallet: add ScriptHash() to Account
It allows to simplify a lot of code and avoid getting a PrivateKey in some
cases.
2022-09-02 14:43:34 +03:00
Roman Khimov
8d33206bb8 *: don't get private key from account if just public one is needed
Add PublicKey() API to the Account and use it as appropriate, avoid creating
additional references to the private key.
2022-09-02 14:43:28 +03:00
Roman Khimov
ac5c609063 core: add a bit more data into NVB errors
It's not always obvious what they mean and the NVB value is.
2022-08-29 15:08:37 +03:00
Roman Khimov
9916832e2e core: set Tx for interop.Context when verifying witnesses
Notary contract uses it in the verification context and it's not harmful to
have it always be there when it's there.
2022-08-22 13:38:43 +03:00
Roman Khimov
eeeb0f6f0e core: accept two-side channels for sub/unsub, read on unsub
Blockchain's notificationDispatcher sends events to channels and these
channels must be read from. Unfortunately, regular service shutdown procedure
does unsubscription first (outside of the read loop) and only then drains the
channel. While it waits for unsubscription request to be accepted
notificationDispatcher can try pushing more data into the same channel which
will lead to a deadlock. Reading in the same method solves this, any number of
events can be pushed until unsub channel accepts the data.
2022-08-19 22:08:40 +03:00
Roman Khimov
f7c5ab4f43 state: check for array length in (*Contract).FromStackItem
Panicing here is not appropriate.
2022-08-16 17:19:47 +03:00
Roman Khimov
a1a5db8fcd state: add more convenient method to get native contract hashes 2022-08-15 10:54:29 +03:00
Anna Shaleva
94e84f0364 core: fix native Management's hasMethod signature
Affects states, see
28ab45a6ec/src/Neo/SmartContract/Native/ContractManagement.cs (L155).
2022-08-11 16:10:01 +03:00
Roman Khimov
bc3bffea53 native: fix oracle.finish reentrancy bug
See neo-project/neo#2795.
2022-08-10 19:02:36 +03:00
Roman Khimov
8d170a1eb8 native: add Oracle.finish reentrancy test 2022-08-10 19:02:36 +03:00