Mostly it's about Go 1.22+ syntax with ranging over integers, but it also
prefers ranging over slices where possible (it makes code a little better to
read).
Notice that we have a number of dangerous loops where slices are mutated
during loop execution, many of these can't be converted since we need proper
length evalutation at every iteration.
Signed-off-by: Roman Khimov <roman@nspcc.ru>
It's possible for transaction to include block hash into Conflicts
attribure. If so, then we must not remove block executable record while
cleaning transation's conflict records.
This commit is a direct consequence of
e6ceee0f230a21c87006a9297636be29c0d8ea47. Ref. #3427.
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Conflicts-related code contains more and more these magic numbers, and
there's no good in it even if all the usages are commented. This
approach produces bugs like #3426.
No functional changes, just a refactoring.
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Conflict record stub has value of 5 bytes length: 1 byte for
storage.ExecTransaction prefix and 4 bytes for the block index LE. This
scheme was implemented in #3138, and this commit should be a part of
this PR.
Also, transaction.DummyVersion is removed since it's unused anymore.
Close#3426. The reason of `failed to locate application log: EOF` error
during genesis AER request is in the following: genesis executable was
overwritten by conflict record stub produced by transaction
0x289c235dcdab8be7426d05f0fbb5e86c619f81481ea136493fa95deee5dbb7cc (ref.
#3427). As a consequence, an attempt to decode transaction AER was
initited, but conflict record scheme was changed in #3138.
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Transaction
0x289c235dcdab8be7426d05f0fbb5e86c619f81481ea136493fa95deee5dbb7cc is
already on mainnet at block 5272006 and we can't do anything with it.
This transaction has genesis block hash in Conflicts attribute. It leads
to the following consequences:
1. Genesis block executable record is overwritten by conflict record
stub. Genesis block can't be retrieved anymore. This bug is described
in #3427.
2. Somehow this transaction has passed verification on NeoGo CN without
any warnings:
```
Apr 24 16:12:30 kangra neo-go[2453907]: 2024-04-24T16:12:30.865+0300 INFO initializing dbft {"height": 5272006, "view": 0, "index": 6, "role": "Backup"}
Apr 24 16:12:31 kangra neo-go[2453907]: 2024-04-24T16:12:31.245+0300 INFO persisted to disk {"blocks": 1, "keys": 37, "headerHeight": 5272005, "blockHeight": 5272005, "took": "14.548903ms"}
Apr 24 16:12:34 kangra neo-go[2453907]: 2024-04-24T16:12:34.977+0300 ERROR can't add SV-signed state root {"error": "stateroot mismatch at block 5272005: 9d5f95784f26c862d6f889f213aad1e3330611880c02330e88db8802c750aa46 vs d25304d518645df725014897d13bbf023919928e79074abcea48f31cf9f32a25"}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.820+0300 INFO received PrepareRequest {"validator": 5, "tx": 1}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.821+0300 INFO sending PrepareResponse {"height": 5272006, "view": 0}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.827+0300 INFO received PrepareResponse {"validator": 4}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.830+0300 INFO received PrepareResponse {"validator": 3}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.875+0300 INFO received PrepareResponse {"validator": 2}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.878+0300 INFO sending Commit {"height": 5272006, "view": 0}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.879+0300 INFO received Commit {"validator": 4}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.881+0300 INFO received PrepareResponse {"validator": 0}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.881+0300 INFO received Commit {"validator": 3}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.906+0300 INFO received Commit {"validator": 0}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.907+0300 INFO received PrepareResponse {"validator": 1}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.915+0300 INFO received Commit {"validator": 1}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.915+0300 INFO approving block {"height": 5272006, "hash": "6b111519537343ce579d04ccad71c43318b12c680d0f374dfcd466aa22643fb6", "tx_count": 1, "merkle": "ccb7dbe5ee5da93f4936a11e48819f616ce8b5fbf0056d42e78babcd5d239c28", "prev": "12ad6cc5d0cd357b9fc9fb0c1a016ba8014d3cdd5a96818598e6a40a1a4a2a21"}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.917+0300 WARN contract invocation failed {"tx": "289c235dcdab8be7426d05f0fbb5e86c619f81481ea136493fa95deee5dbb7cc", "block": 5272006, "error": "at instruction 86 (ASSERT): ASSERT failed"}
Apr 24 16:12:45 kangra neo-go[2453907]: 2024-04-24T16:12:45.950+0300 INFO initializing dbft {"height": 5272007, "view": 0, "index": 6, "role": "Primary"}
Apr 24 16:12:46 kangra neo-go[2453907]: 2024-04-24T16:12:46.256+0300 INFO persisted to disk {"blocks": 1, "keys": 67, "headerHeight": 5272006, "blockHeight": 5272006, "took": "16.576594ms"}
```
And thus, we must treat this transaction as valid for this behaviour
to be reproducable.
This commit contains two fixes:
1. Do not overwrite block executable records by conflict record stubs.
If some transaction conflicts with block, then just skip the conflict
record stub for this attribute since it's impossible to create
transaction with the same hash.
2. Do not fail verification for those transactions that have Conflicts
attribute with block hash inside. This one is controversial, but we
have to adjust this code to treat already accepted transaction as
valid.
Close#3427.
The transaction itself:
```
{
"id" : 1,
"jsonrpc" : "2.0",
"result" : {
"attributes" : [
{
"height" : 0,
"type" : "NotValidBefore"
},
{
"hash" : "0x1f4d1defa46faa5e7b9b8d3f79a06bec777d7c26c4aa5f6f5899a291daa87c15",
"type" : "Conflicts"
}
],
"blockhash" : "0xb63f6422aa66d4fc4d370f0d682cb11833c471adcc049d57ce4373531915116b",
"blocktime" : 1713964365700,
"confirmations" : 108335,
"hash" : "0x289c235dcdab8be7426d05f0fbb5e86c619f81481ea136493fa95deee5dbb7cc",
"netfee" : "237904",
"nonce" : 0,
"script" : "CxAMFIPvkoyXujYCRmgq9qEfMJQ4wNveDBSD75KMl7o2AkZoKvahHzCUOMDb3hTAHwwIdHJhbnNmZXIMFPVj6kC8KD1NDgXEjqMFs/Kgc0DvQWJ9W1I5",
"sender" : "NbcGB1tBEGM5MfhNbDAimvpJKzvVjLQ3jW",
"signers" : [
{
"account" : "0x649ca095e38a790d6c15ff78e0c6175099b428ac",
"scopes" : "None"
},
{
"account" : "0xdedbc03894301fa1f62a68460236ba978c92ef83",
"scopes" : "None"
}
],
"size" : 412,
"sysfee" : "997778",
"validuntilblock" : 5277629,
"version" : 0,
"vmstate" : "FAULT",
"witnesses" : [
{
"invocation" : "DECw8XNuyRg5vPeHxisQXlZ7VYNDxxK4xEm8zwpPyWJSSu+JaRKQxdrlPkXxXj34wc4ZSrZvKICGgPFE0ZHXhLPo",
"verification" : "DCEC+PI2tRSlp0wGwnjRuQdWdI0tBXNS7SlzSBBHFsaKUsdBVuezJw=="
},
{
"invocation" : "DEAxwi97t+rg9RsccOUzdJTJK7idbR7uUqQp0/0/ob9FbuW/tFius3/FOi82PDZtwdhk7s7KiNM/pU7vZLsgIbM0",
"verification" : "DCEDbInkzF5llzmgljE4HSMvtrNgPaz73XO5wgVJXLHNLXRBVuezJw=="
}
]
}
}
```
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
And refactor some code a bit, don't use bytes.Clone where type-specific
helpers may be used instead.
Close#2907.
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
We often use binary.PutUint*, but almost all these cases have preallocated
buffer of the size that matches exactly the desired one and use a single or
a couple of calls to PutUint*. Thus, I don't think that replacing
binary.PutUint* by AppendUint* will make things better for all these usages.
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
During new transaction verification if there's an on-chain conflicting
transaction, we should check the signers of this conflicting transaction.
If the signers intersect with signers of the incoming transaction, then
the conflict is treated as valid and verification for new incoming
transaction should fail. Otherwise, the conflict is treated as the
malicious attack attempt and will not be taken into account;
verification for the new incoming transaction should continue.
This commint implements the scheme described at
https://github.com/neo-project/neo/pull/2818#issuecomment-1632972055,
thanks to @shargon for digging.
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
We're paging these hashes, so we need a previous full page and a current one
plus some cache for various requests. Storing 1M of hashes is 32M of memory
and it grows quickly. It also seriously affects node startup time, most of
what it's doing is reading these hashes, the longer the chain the more time it
needs to do that.
Notice that this doesn't change the underlying DB scheme in any way.
We need to keep the headers information consistent with header batches
and headers. This comit fixes the bug with failing blockchain
initialization on recovering from state reset interrupted after the
second stage (blocks/txs/AERs removal):
```
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go db reset -t --height 83000
2022-11-20T16:28:29.437+0300 INFO MaxValidUntilBlockIncrement is not set or wrong, using default value {"MaxValidUntilBlockIncrement": 5760}
2022-11-20T16:28:29.440+0300 INFO restoring blockchain {"version": "0.2.6"}
failed to create Blockchain instance: could not initialize blockchain: could not get header 1898cd356a4a2688ed1c6c7ba1fd6ba7d516959d8add3f8dd26232474d4539bd: key not found
```
Turns out, it's almost always allocating because we're mostly dealing with
small integers while the buffer size is calculated in 8-byte chunks here, so
preallocated buffer is always insufficient.
name old time/op new time/op delta
ToPreallocatedBytes-8 28.5ns ± 7% 19.7ns ± 5% -30.72% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ToPreallocatedBytes-8 16.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ToPreallocatedBytes-8 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
Fix StorageItem reuse at the same time. We don't copy when getting values from
the storage, but we don when we're putting them, so buffer reuse could corrupt
old values.
Fix the following linter warning:
```
pkg/core/dao/dao.go:101:7 govet copylocks: assignment copies lock value to *d: github.com/nspcc-dev/neo-go/pkg/core/dao.Simple contains sync.RWMutex
```
The only user of (*Block).Trim() is in DAO and it already has a nice buffer
usually, so creating another one makes no sense. It also simplifies error
handling a lot.
We don't have a need to iterate over them at the moment, but since we're
changing the DB format in the next release anyway let's add this ability also,
just in case.
It couldn't be done previously with two maps and mixed storage, but now all of
the storage changes are located in a single map, so it's trivial to do exact
slice allocations and avoid string->[]byte conversions.
Private DAO is only used in a single thread which means we can safely reuse
key/data buffers most of the time and handle it all in DAO.
Doesn't affect any benchmarks.
Most of the time we don't need locking on the higher-level stores and we drop
them after Persist, so that's what private MemCachedStore is for.
It doesn't improve things in any noticeable way, some ~1% can be observed in
neo-bench under various loads and even less than that in chain processing. But
it seems to be a bit better anyway (less allocations, less locks).
They never return errors, so their interface should reflect that. This allows
to remove quite a lot of useless and never tested code.
Notice that Get still does return an error. It can be made not to do that, but
usually we need to differentiate between successful/unsuccessful accesses
anyway, so this doesn't help much.
