Commit graph

5785 commits

Author SHA1 Message Date
Evgeniy Stratonikov
c212b2be95 util: allow to marshal Uint160 in YAML
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2022-02-21 10:38:20 +03:00
Roman Khimov
9224f57323
Merge pull request #2365 from nspcc-dev/vm/replace-cli
vm CLI: replace ishell with urfave/cli
2022-02-18 19:30:39 +03:00
AnnaShaleva
c35950206a vm: add test to check CLI UTF-8 compatibility 2022-02-18 19:20:50 +03:00
Anna Shaleva
c182141624 vm: fix linter issue
printf: `fmt.Fprintln` arg list ends with redundant newline (govet)
2022-02-18 19:20:43 +03:00
Roman Khimov
4cc1413805
Merge pull request #2366 from nspcc-dev/tiny-dao-deduplication
Some DAO deduplication
2022-02-18 19:10:48 +03:00
Anna Shaleva
2501ce3cb1 vm: adjust run description for VM CLI
With urfave/cli method name can be `help`.
2022-02-18 16:34:58 +03:00
Anna Shaleva
d04d55708c vm: setup autocompleter 2022-02-18 16:34:58 +03:00
Anna Shaleva
e8ec820d2e vm: add unload command
When execution fails it's the only way to change the prompt showing the
last executed instruction. Example:

```
NEO-GO-VM > loadgo examples/engine/engine.go
READY: loaded 117 instructions
NEO-GO-VM 0 > run
Error: at instruction 3 (SYSCALL): failed to invoke syscall 805851437: syscall not found
NEO-GO-VM 8 > parse
Error: missing argument
NEO-GO-VM 8 > parse 123
Integer to Hex			7b
Integer to Base64		ew==
String to Hex			313233
String to Base64		MTIz

NEO-GO-VM 8 > unload
NEO-GO-VM > exit

```
2022-02-18 16:34:58 +03:00
Anna Shaleva
563c3a4baa vm: replace ishell with urfave/cli
Use github.com/chzyer/readline for readline capabilities (including
history and ANSI escape sequences handling).
2022-02-18 16:34:50 +03:00
Roman Khimov
7d6f087337 storage: drop (KeyPrefix).Bytes() API
It allocates and most of the time we can avoid that.
2022-02-18 15:19:57 +03:00
Roman Khimov
7223caf369 dao: improve PutCurrentHeader logic
Move serialization out of the core.
2022-02-18 15:05:25 +03:00
Roman Khimov
522229d731 storage: drop AppendPrefix/AppendPrefixInt APIs
We're not using them anymore and they allocate.
2022-02-18 14:59:59 +03:00
Roman Khimov
d2db58d748 dao: move header hash store logic out of the core
Which allows for more efficient buffer use along the way.
2022-02-18 14:54:05 +03:00
Roman Khimov
de2579ec07 dao: put contract IDs into keys using big endianness
We don't have a need to iterate over them at the moment, but since we're
changing the DB format in the next release anyway let's add this ability also,
just in case.
2022-02-18 14:38:51 +03:00
Roman Khimov
600da6909c storage: put uint32 into keys using in big endianness
Which allows to iterate over the contents easily.
2022-02-18 14:35:17 +03:00
Roman Khimov
1ca918e631 dao: delay buffer creation until it's needed
Verification contexts don't ever touch the storage, so these allocations can
be avoided for them.
2022-02-18 14:24:45 +03:00
Roman Khimov
e864768c88 dao: simplify NewPrivate 2022-02-18 14:18:56 +03:00
Roman Khimov
5402e654d1 core: don't create useless DAO layer in GetTestVM
We're already wrapping in interop.NewContext.
2022-02-18 14:12:44 +03:00
Roman Khimov
b60d4ff191 dao: deduplicate header->KV conversion 2022-02-18 14:12:44 +03:00
Roman Khimov
d8cf879499 dao: deduplicate DeleteBlock, no functional changes 2022-02-18 14:12:44 +03:00
Roman Khimov
baa05bef43
Merge pull request #2364 from nspcc-dev/botch
storage: speedups and simplifications
2022-02-18 14:11:40 +03:00
Roman Khimov
f80680187e storage: expose private storage map for more efficient MPT batch
It couldn't be done previously with two maps and mixed storage, but now all of
the storage changes are located in a single map, so it's trivial to do exact
slice allocations and avoid string->[]byte conversions.
2022-02-17 23:41:10 +03:00
Roman Khimov
7dc8fc443f dao: simplify buffer management for private DAO
Private DAO is only used in a single thread which means we can safely reuse
key/data buffers most of the time and handle it all in DAO.

Doesn't affect any benchmarks.
2022-02-17 22:27:39 +03:00
Roman Khimov
9bfb3357f2 storage: add "private" mode to MemCachedStore
Most of the time we don't need locking on the higher-level stores and we drop
them after Persist, so that's what private MemCachedStore is for.

It doesn't improve things in any noticeable way, some ~1% can be observed in
neo-bench under various loads and even less than that in chain processing. But
it seems to be a bit better anyway (less allocations, less locks).
2022-02-17 22:27:39 +03:00
Roman Khimov
aefb26255a dao: drop DAO interface
It's a remnant from the days when we had Simple and Cached DAO
implementations, now it makes zero sense.
2022-02-16 18:24:20 +03:00
Roman Khimov
9d2ef775cf storage: simplify (*MemCachedStore).Put/Delete interface
They never return errors, so their interface should reflect that. This allows
to remove quite a lot of useless and never tested code.

Notice that Get still does return an error. It can be made not to do that, but
usually we need to differentiate between successful/unsuccessful accesses
anyway, so this doesn't help much.
2022-02-16 18:24:20 +03:00
Roman Khimov
be24bf6412 storage: drop Put and Delete from Store interface
It's only changed with PutChangeSet, single KV operations are handled by
MemCachedStore.
2022-02-16 18:24:20 +03:00
Roman Khimov
017795c9c1 storage: completely remove MemoryBatch
If you need something like that, just wrap another MemCachedStore layer around
it.
2022-02-16 16:13:12 +03:00
Roman Khimov
17a43b19e0 storage: remove Batch from Store
We never use it for real underlying stores, so these implementations are
useless (everything goes though PutChangeSet now).
2022-02-16 15:55:48 +03:00
Roman Khimov
35bdfc5eca storage: use two maps for MemoryStore
Simple and dumb as it is, this allows to separate contract storage from other
things and dramatically improve Seek() time over storage (even though it's
still unordered!) which in turn improves block processing speed.

        LevelDB             LevelDB (KeepOnlyLatest)  BoltDB              BoltDB (KeepOnlyLatest)
Master  real    16m27,936s  real    10m9,440s         real    16m39,369s  real    8m1,227s
        user    20m12,619s  user    26m13,925s        user    18m9,162s   user    18m5,846s
        sys     2m56,377s   sys     1m32,051s         sys     9m52,576s   sys     2m9,455s

2 maps  real    10m49,495s  real    8m53,342s         real    11m46,204s  real    5m56,043s
        user    14m19,922s  user    24m6,225s         user    13m25,691s  user    15m4,694s
        sys     1m53,021s   sys     1m23,006s         sys     4m31,735s   sys     2m8,714s

neo-bench performance is mostly unaffected, ~0.5% for 1-1 test and 4% for
10K-10K test both fall within regular test error range.
2022-02-16 15:55:48 +03:00
Roman Khimov
0767120e8a
Merge pull request #2363 from nspcc-dev/drop-old-nep-transfers
core: remove stale transfer entries, fix #2096
2022-02-16 15:25:10 +03:00
Roman Khimov
b6a4947cfd storage: drop STAccount prefix
It's unused for a long time.
2022-02-16 13:06:57 +03:00
Roman Khimov
54bc603831 core: remove old storage items synchronously during jump
There won't be a lot of them and GC experience we have shows that iterating
over 1-2M of entries is not a huge problem for DBs we have.
2022-02-16 13:03:13 +03:00
Roman Khimov
805d81e441 core: drop old transfer data during state jump 2022-02-16 12:12:03 +03:00
Roman Khimov
3a2b800e5f core: remove stale transfer entries, fix #2096
Initially I thought of doing it in the next persist cycle, but testing shows
that it needs just ~2-5% of the time MPT GC does, so doing it in the same
cycle doesn't affect anything.
2022-02-15 15:51:03 +03:00
Roman Khimov
8216500fd7
Merge pull request #2360 from nspcc-dev/gc-optimizations
GC optimizations
2022-02-14 17:47:51 +03:00
Roman Khimov
c4b49a2d52 storage: deduplicate storage closing in tests 2022-02-14 17:29:21 +03:00
Roman Khimov
261552516b stateroot: use a better word for logs 2022-02-14 17:29:21 +03:00
Roman Khimov
ccdda21718 stateroot: add and use DataMPTAux for auxiliary data
Use DataMPT for nodes only, otherwise with 1M blocks with have 1M
height-stateroot mapping entries that our GC has to iterate over for no
reason.
2022-02-14 17:29:21 +03:00
Roman Khimov
ad606101c7 storage: add SeekGC interface for GC
It's very special, single-purpose thing, but it improves cumulative time spent
in GC by ~10% for LevelDB and by ~36% for BoltDB during 1050K mainnet chain
processing. While the overall chain import time doesn't change in any
noticeable way (~1%), I think it's still worth it, for machines with slower
disks the difference might be more noticeable.
2022-02-14 17:29:21 +03:00
Roman Khimov
51b804ab0e storage: generalize Level/Bolt seek implementations
Too much in common. Just refactoring. no functional changes.
2022-02-12 23:09:37 +03:00
Roman Khimov
a5f8b8870a storage: use Update for changeset processing
Batch is only relevant in multithreaded context, internally it'll do some
magic and use the same locking/updating Update does, so it makes little sense
for us. This doesn't change benchmarks in any noticeable way.
2022-02-11 16:48:35 +03:00
Roman Khimov
6380647770
Merge pull request #2354 from nspcc-dev/mpt-value-based-gc
Value-based MPT GC
2022-02-11 16:25:08 +03:00
Roman Khimov
373fce54e6 config: conflict P2PStateExchangeExtensions/KeepOnlyLatestState
They don't make sense together, for P2P state exchange to be possible we need
a set of MPTs.
2022-02-11 14:19:54 +03:00
Roman Khimov
c5f1e2fbcd mpt: don't allocate for every key
Microoptimization.
2022-02-11 14:19:54 +03:00
Roman Khimov
423c7883b8 core: implement basic GC for value-based storage scheme
The key idea here is that even though we can't ensure MPT code won't make the
node active again we can order the changes made to the persistent store in
such a way that it practically doesn't matter. What happens is:
 * after persist if it's time to collect our garbage we do it synchronously
   right in the same thread working the underlying persistent store directly
 * all the other node code doesn't see much of it, it works with bc.dao or
   layers above it
 * if MPT doesn't find some stale deactivated node in the storage it's OK,
   it'll recreate it in bc.dao
 * if MPT finds it and activates it, it's OK too, bc.dao will store it
 * while GC is being performed nothing else changes the persistent store
 * all subsequent bc.dao persists only happen after the GC is completed which
   means that any changes to the (potentially) deleted nodes have a priority,
   it's OK for GC to delete something that'll be recreated with the next
   persist cycle

Otherwise it's a simple scheme with node status/last active height stored in
the value. Preliminary tests show that it works ~18% worse than the simple
KeepOnlyLatest scheme, but this seems to be the best result so far.

Fixes #2095.
2022-02-11 14:19:54 +03:00
Roman Khimov
075fd05bfc
Merge pull request #2356 from nspcc-dev/rpc-big-numbers
Allow to use big integers in RPC
2022-02-11 13:24:35 +03:00
Evgeniy Stratonikov
037076f2ee rpc/request: return error from ExpandArrayIntoScript if any
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:48:55 +03:00
Evgeniy Stratonikov
c72b3f2176 vm/emit: check big integer size
Fix a bug where big integer could be emitted as 0 without reporting an
error. Also, add tests.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:48:55 +03:00
Evgeniy Stratonikov
0394a79ef8 rpc: support big integers as request parameters
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:48:54 +03:00