Instead of tick-tocking with sync/async and having an unpredictable data
set we can just try to check for the real amount of keys that be processed
by the underlying DB. Can't be perfect, but still this adds some hard
limit to the amount of in-memory data. It's also adaptive, slower machines
will keep less and faster machines will keep more.
This gives almost perfect 4s cycles for mainnet BoltDB with no tail cutting,
it makes zero sense to process more blocks since we're clearly DB-bound:
2025-01-15T11:35:00.567+0300 INFO persisted to disk {"blocks": 1469, "keys": 40579, "headerHeight": 5438141, "blockHeight": 5438140, "velocity": 9912, "took": "4.378939648s"}
2025-01-15T11:35:04.699+0300 INFO persisted to disk {"blocks": 1060, "keys": 39976, "headerHeight": 5439201, "blockHeight": 5439200, "velocity": 9888, "took": "4.131985438s"}
2025-01-15T11:35:08.752+0300 INFO persisted to disk {"blocks": 1508, "keys": 39658, "headerHeight": 5440709, "blockHeight": 5440708, "velocity": 9877, "took": "4.052347569s"}
2025-01-15T11:35:12.807+0300 INFO persisted to disk {"blocks": 1645, "keys": 39565, "headerHeight": 5442354, "blockHeight": 5442353, "velocity": 9864, "took": "4.05547743s"}
2025-01-15T11:35:17.011+0300 INFO persisted to disk {"blocks": 1472, "keys": 39519, "headerHeight": 5443826, "blockHeight": 5443825, "velocity": 9817, "took": "4.203258142s"}
2025-01-15T11:35:21.089+0300 INFO persisted to disk {"blocks": 1345, "keys": 39529, "headerHeight": 5445171, "blockHeight": 5445170, "velocity": 9804, "took": "4.078297579s"}
2025-01-15T11:35:25.090+0300 INFO persisted to disk {"blocks": 1054, "keys": 39326, "headerHeight": 5446225, "blockHeight": 5446224, "velocity": 9806, "took": "4.000524899s"}
2025-01-15T11:35:30.372+0300 INFO persisted to disk {"blocks": 1239, "keys": 39349, "headerHeight": 5447464, "blockHeight": 5447463, "velocity": 9744, "took": "4.281444939s"}
2× can be considered, but this calculation isn't perfect for low number of
keys, so somewhat bigger tolerance is preferable for now. Overall it's not
degrading performance, my mainnet/bolt run was even 8% better with this.
Fixes#3249, we don't need any option this way.
Fixes#3783 as well, it no longer OOMs in that scenario. It however can OOM in
case of big GarbageCollectionPeriod (like 400K), but this can't be solved easily.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
RemoveUntraceableBlocks without RemoveUntraceableHeaders is still a valid
configuration and removeOldTransfers() only checks for gcBlockTimes now
for timestamps, so they should always be added. This fixes
2025-01-13T23:28:57.340+0300 ERROR failed to get block timestamp transfer GC {"time": "1.162µs", "index": 20000}
for RemoveUntraceableBlocks-only configurations.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
The intention here is to reduce the amount of in-flight changes and prevent
OOM. It doesn't matter what we're doing, persisting or collecting garbage,
what matters is that we're behind the schedule of regular persist cycle.
Refs. #3783.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
It doesn't change anything logically, but transfer GC needs to have current
block timestamp for its logic and if we're delaying it with MPT GC it can
more often fail to obtain it:
2025-01-13T16:15:18.311+0300 ERROR failed to get block timestamp transfer GC {"time": "1.022µs", "index": 20000}
It's not critical, this garbage can still be collected on the next run, but
we better avoid this anyway.
Refs. #3783.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
Solves two problems:
* inability to estimate GAS needed for registerCandidate in a regular way
because of its very high fee (more than what normal RPC servers allow)
* inability to have MaxBlockSystemFee lower than the registration price
which is very high on its own (more than practically possible to execute)
See https://github.com/neo-project/neo/issues/3552.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
Transfer data is timestamp-based, previously it always had and used headers,
no we can go via a small cache (we don't want it to grow or be stored forever).
Otherwise it's unable to do the job:
2024-12-13T12:55:15.056+0300 ERROR failed to find block header for transfer GC {"time": "19.066µs", "error": "key not found"}
Signed-off-by: Roman Khimov <roman@nspcc.ru>
It's there since 423c7883b8, but looks like it
never changed anything, the same thing is done six lines above and tgtBlock is
not changed since.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
It's important to have the same chain configuration for all tests.
Otherwise, a mismatched hardfork configuration is used to dump/restore
the basic chain.
Close#3681
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
Function doesn't make much sense here. The change is rather trivial and this
is not expected to be used by external code, so no deprecation.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
Differentiate released and stable ones from test-only not-yet-ready and such.
Enabled only stable ones by default to avoid surprises in private networks
when some beta hardfork is made available with some node release.
Fixes#3719.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
Technically, we can always buildHFSpecificMD() if contract is active, but we
just optimize the build out in some cases. switch slightly obfuscates this
and requires having the call in two branches.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
These conditions are about filtering methods out. A method is excluded if its
ActiveFrom is strictly higher than the current HF, since if it's the same then
it should be present. Otherwise contract is broken at the height of this
particular HF.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
```
pkg/vm/stackitem/json_test.go:11 gofmt File is
not `gofmt`-ed with `-s` `-r 'interface{} -> any'`
pkg/core/native/native_test/cryptolib_test.go:471 gofmt File is
not `gofmt`-ed with `-s` `-r 'interface{} -> any'`
pkg/rpcclient/nns/contract_test.go:585 gofmt File is
not `gofmt`-ed with `-s` `-r 'interface{} -> any'`
```
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>
It took some time to understand why changing a regular `for` to a `range`
one leads to behavior changes; let it be more clear and explicit. Also, a
correct code is always better than a correct code with `nolint`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Mostly it's about Go 1.22+ syntax with ranging over integers, but it also
prefers ranging over slices where possible (it makes code a little better to
read).
Notice that we have a number of dangerous loops where slices are mutated
during loop execution, many of these can't be converted since we need proper
length evalutation at every iteration.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
It's slightly less efficient (all comparisons are always made), but for
strings/ints it's negligible performance difference, while the code looks a
tiny bit better.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
"Compare" is almost a standard one now, although math/big uses Cmp for historic
reasons (keys.PublicKey does so too). This also fixes Fixed8 since int64 to int
conversion is lossy.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
