neoneo-go/pkg/core
Anna Shaleva bdc42cd595 core: reset blocks, txs and AERs in several stages
Sometimes it can be hard to persist all changes at ones, the process
can take almost all RAM and a lot of time. Here's the example of reset
for mainnet from 2.4M to 1:
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -m --height 1
2022-11-20T17:16:48.236+0300	INFO	MaxBlockSize is not set or wrong, setting default value	{"MaxBlockSize": 262144}
2022-11-20T17:16:48.236+0300	INFO	MaxBlockSystemFee is not set or wrong, setting default value	{"MaxBlockSystemFee": 900000000000}
2022-11-20T17:16:48.237+0300	INFO	MaxTransactionsPerBlock is not set or wrong, using default value	{"MaxTransactionsPerBlock": 512}
2022-11-20T17:16:48.237+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T17:16:48.240+0300	INFO	restoring blockchain	{"version": "0.2.6"}
2022-11-20T17:16:48.297+0300	INFO	initialize state reset	{"target height": 1}
2022-11-20T17:16:48.300+0300	INFO	trying to reset blocks, transactions and AERs
2022-11-20T17:19:29.313+0300	INFO	blocks, transactions ans AERs are reset	{"took": "2m41.015126493s", "keys": 3958420}
...
```
To avoid OOM killer, split blocks reset into multiple stages. It increases
operation time due to intermediate DB persists, but makes things cleaner, the
result for almost the same DB height with the new approach:
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -m --height 1
2022-11-20T17:39:42.023+0300	INFO	MaxBlockSize is not set or wrong, setting default value	{"MaxBlockSize": 262144}
2022-11-20T17:39:42.023+0300	INFO	MaxBlockSystemFee is not set or wrong, setting default value	{"MaxBlockSystemFee": 900000000000}
2022-11-20T17:39:42.023+0300	INFO	MaxTransactionsPerBlock is not set or wrong, using default value	{"MaxTransactionsPerBlock": 512}
2022-11-20T17:39:42.023+0300	INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2022-11-20T17:39:42.026+0300	INFO	restoring blockchain	{"version": "0.2.6"}
2022-11-20T17:39:42.071+0300	INFO	initialize state reset	{"target height": 1}
2022-11-20T17:39:42.073+0300	INFO	trying to reset blocks, transactions and AERs
2022-11-20T17:40:11.735+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 1, "took": "29.66363737s", "keys": 210973}
2022-11-20T17:40:33.574+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 2, "took": "21.839208683s", "keys": 241203}
2022-11-20T17:41:29.325+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 3, "took": "55.750698386s", "keys": 250593}
2022-11-20T17:42:12.532+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 4, "took": "43.205892757s", "keys": 321896}
2022-11-20T17:43:07.978+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 5, "took": "55.445398156s", "keys": 334822}
2022-11-20T17:43:35.603+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 6, "took": "27.625292032s", "keys": 317131}
2022-11-20T17:43:51.747+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 7, "took": "16.144359017s", "keys": 355832}
2022-11-20T17:44:05.176+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 8, "took": "13.428733899s", "keys": 357690}
2022-11-20T17:44:32.895+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 9, "took": "27.718548783s", "keys": 393356}
2022-11-20T17:44:51.814+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 10, "took": "18.917954658s", "keys": 366492}
2022-11-20T17:45:07.208+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 11, "took": "15.392642196s", "keys": 326030}
2022-11-20T17:45:18.776+0300	INFO	intermediate batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 12, "took": "11.568255716s", "keys": 299884}
2022-11-20T17:45:25.862+0300	INFO	last batch of removed blocks, transactions and AERs is persisted	{"batches persisted": 13, "took": "7.086079594s", "keys": 190399}
2022-11-20T17:45:25.862+0300	INFO	blocks, transactions ans AERs are reset	{"took": "5m43.791214084s", "overall persisted keys": 3966301}
...
```
2022-11-22 11:53:39 +03:00
..
block block: JSONize tx-less block as [] instead of null 2022-07-21 13:15:31 +03:00
chaindump core: move chain dump test into its own package 2022-06-08 15:28:08 +03:00
dao core: reset block headers together with header height info 2022-11-22 11:53:39 +03:00
fee vm: add MODMUL and MODPOW opcodes 2022-05-12 14:25:14 +03:00
interop cli: use custom logger to filter out runtime.Log messages 2022-10-07 15:57:33 +03:00
mempool [#2442] English Check 2022-05-04 19:48:27 +03:00
mempoolevent [#2442] English Check 2022-05-04 19:48:27 +03:00
mpt golangci: enable/fix misspell 2022-09-02 18:36:26 +03:00
native native: ignore decoding errors during cache init 2022-11-16 12:00:28 +03:00
state core: fix typo 2022-11-10 17:58:06 +03:00
stateroot core: allow to reset blockchain state 2022-11-10 18:08:17 +03:00
statesync rpcclient: fix WS-client unsubscription process 2022-11-16 23:44:30 +03:00
storage core: allow to reset blockchain state 2022-11-10 18:08:17 +03:00
test_data core: refactor helper test contracts generation 2022-03-30 11:46:29 +03:00
transaction core: add example to VUB comment 2022-10-06 10:32:18 +03:00
basic_chain_test.go *: apply go 1.19 formatter heuristics 2022-08-09 15:37:52 +03:00
bench_test.go *: reuse smartcontract package to create standard entry scripts 2022-07-26 12:19:49 +03:00
blockchain.go core: reset blocks, txs and AERs in several stages 2022-11-22 11:53:39 +03:00
blockchain_core_test.go core: allow to reset blockchain state 2022-11-10 18:08:17 +03:00
blockchain_neotest_test.go native: ignore decoding errors during cache init 2022-11-16 12:00:28 +03:00
doc.go *: apply go 1.19 formatter heuristics 2022-08-09 15:37:52 +03:00
helper_test.go core: rebase core tests onto neotest 2022-03-30 19:00:53 +03:00
interops.go core: move storage-related interop code into the storage package 2022-06-08 22:26:24 +03:00
native_designate_test.go core: add some comment for TestDesignate_DesignateAsRole 2022-06-08 23:11:14 +03:00
prometheus.go stateroot: move state-root related logic to core/stateroot 2021-03-09 13:48:29 +03:00
util.go core: export CreateGenesisBlock 2022-06-08 18:20:34 +03:00
util_test.go core: export CreateGenesisBlock 2022-06-08 18:20:34 +03:00