Commit graph

4029 commits

Author SHA1 Message Date
Roman Khimov
870fd024c9
Merge pull request #2367 from nspcc-dev/rpc/thread-safe
rpc: take care of RPC clients
2022-02-24 20:11:15 +03:00
Anna Shaleva
1f255e756f rpc: add a note to RPC clients about thread-safeness 2022-02-24 19:01:21 +03:00
Anna Shaleva
2896c0a83a rpc: add test for concurrent WSClient access and request IDs generation 2022-02-24 19:01:21 +03:00
AnnaShaleva
0d8723527c rpc: refactor WSClient initialisation
Fix the following linter warning:
```
pkg/rpc/client/wsclient.go:99:18  govet  copylocks: literal copies lock value from *cl: github.com/nspcc-dev/neo-go/pkg/rpc/client.Client contains sync.RWMutex
```
2022-02-24 19:01:21 +03:00
AnnaShaleva
8991ee91cd rpc: make RPC WSClient thread-safe
Add ability to use unique request IDs for RPC requests.
2022-02-24 19:01:12 +03:00
Roman Khimov
b65ab011fe
Merge pull request #2349 from nspcc-dev/codegen-contract
cli/smartcontract: generate bindings to an existing contract
2022-02-24 15:51:33 +03:00
Anna Shaleva
0357d9f4f4 smartcontract: add hash check to paramcontext
Close #2344.
2022-02-24 10:21:08 +03:00
Evgeniy Stratonikov
da03d895de cli/smartcontract: generate bindings to an existing contract
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2022-02-23 15:40:06 +03:00
AnnaShaleva
5b2e88b916 rpc: protect supscriptions of RPC WSClient from concurrent access 2022-02-22 15:40:08 +03:00
Anna Shaleva
d77f188d10 rpc: update native contract hashes inside (c *Client) GetNativeContracts() 2022-02-22 15:39:42 +03:00
AnnaShaleva
408f6b050e rpc: make RPC Client thread-safe 2022-02-22 15:37:00 +03:00
AnnaShaleva
0092330fe1 rpc: carefully review places where Client.cache.initDone is used
1. Keep initDone check only for the places where cache is directly accessed.
   We don't need to check it in other places, otherwise we have a mess of
   duplicating checks.
2. Fix bug in code related to block deserialisation. There's no magic, so
   checking that initialisation is done is not enough for proper block
   deserialisation. We need to manually fill StateRootEnabled field.
3. Since transaction doesn't need network magic to compute its hash, we don't
   need to perform Client initialisation before transaction-related requests.
4. Check that cache is initialised before accessing network magic.
5. Refactor the way Policy contract hash is fetched for Client requests.
   We don't really need Client initialisation for that, it's OK to fetch Policy
   hash on-the-fly.
2022-02-22 12:52:14 +03:00
AnnaShaleva
7c1862a9ac rpc: move all RPC client's cached fields to a separate cache structure
No functional changes.
2022-02-21 18:16:49 +03:00
Evgeniy Stratonikov
4aab85ef51 callflag: allow to marshal call flags in YAML
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2022-02-21 10:43:35 +03:00
Evgeniy Stratonikov
c212b2be95 util: allow to marshal Uint160 in YAML
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2022-02-21 10:38:20 +03:00
Roman Khimov
9224f57323
Merge pull request #2365 from nspcc-dev/vm/replace-cli
vm CLI: replace ishell with urfave/cli
2022-02-18 19:30:39 +03:00
AnnaShaleva
c35950206a vm: add test to check CLI UTF-8 compatibility 2022-02-18 19:20:50 +03:00
Anna Shaleva
c182141624 vm: fix linter issue
printf: `fmt.Fprintln` arg list ends with redundant newline (govet)
2022-02-18 19:20:43 +03:00
Anna Shaleva
2501ce3cb1 vm: adjust run description for VM CLI
With urfave/cli method name can be `help`.
2022-02-18 16:34:58 +03:00
Anna Shaleva
d04d55708c vm: setup autocompleter 2022-02-18 16:34:58 +03:00
Anna Shaleva
e8ec820d2e vm: add unload command
When execution fails it's the only way to change the prompt showing the
last executed instruction. Example:

```
NEO-GO-VM > loadgo examples/engine/engine.go
READY: loaded 117 instructions
NEO-GO-VM 0 > run
Error: at instruction 3 (SYSCALL): failed to invoke syscall 805851437: syscall not found
NEO-GO-VM 8 > parse
Error: missing argument
NEO-GO-VM 8 > parse 123
Integer to Hex			7b
Integer to Base64		ew==
String to Hex			313233
String to Base64		MTIz

NEO-GO-VM 8 > unload
NEO-GO-VM > exit

```
2022-02-18 16:34:58 +03:00
Anna Shaleva
563c3a4baa vm: replace ishell with urfave/cli
Use github.com/chzyer/readline for readline capabilities (including
history and ANSI escape sequences handling).
2022-02-18 16:34:50 +03:00
Roman Khimov
7d6f087337 storage: drop (KeyPrefix).Bytes() API
It allocates and most of the time we can avoid that.
2022-02-18 15:19:57 +03:00
Roman Khimov
7223caf369 dao: improve PutCurrentHeader logic
Move serialization out of the core.
2022-02-18 15:05:25 +03:00
Roman Khimov
522229d731 storage: drop AppendPrefix/AppendPrefixInt APIs
We're not using them anymore and they allocate.
2022-02-18 14:59:59 +03:00
Roman Khimov
d2db58d748 dao: move header hash store logic out of the core
Which allows for more efficient buffer use along the way.
2022-02-18 14:54:05 +03:00
Roman Khimov
de2579ec07 dao: put contract IDs into keys using big endianness
We don't have a need to iterate over them at the moment, but since we're
changing the DB format in the next release anyway let's add this ability also,
just in case.
2022-02-18 14:38:51 +03:00
Roman Khimov
600da6909c storage: put uint32 into keys using in big endianness
Which allows to iterate over the contents easily.
2022-02-18 14:35:17 +03:00
Roman Khimov
1ca918e631 dao: delay buffer creation until it's needed
Verification contexts don't ever touch the storage, so these allocations can
be avoided for them.
2022-02-18 14:24:45 +03:00
Roman Khimov
e864768c88 dao: simplify NewPrivate 2022-02-18 14:18:56 +03:00
Roman Khimov
5402e654d1 core: don't create useless DAO layer in GetTestVM
We're already wrapping in interop.NewContext.
2022-02-18 14:12:44 +03:00
Roman Khimov
b60d4ff191 dao: deduplicate header->KV conversion 2022-02-18 14:12:44 +03:00
Roman Khimov
d8cf879499 dao: deduplicate DeleteBlock, no functional changes 2022-02-18 14:12:44 +03:00
Roman Khimov
f80680187e storage: expose private storage map for more efficient MPT batch
It couldn't be done previously with two maps and mixed storage, but now all of
the storage changes are located in a single map, so it's trivial to do exact
slice allocations and avoid string->[]byte conversions.
2022-02-17 23:41:10 +03:00
Roman Khimov
7dc8fc443f dao: simplify buffer management for private DAO
Private DAO is only used in a single thread which means we can safely reuse
key/data buffers most of the time and handle it all in DAO.

Doesn't affect any benchmarks.
2022-02-17 22:27:39 +03:00
Roman Khimov
9bfb3357f2 storage: add "private" mode to MemCachedStore
Most of the time we don't need locking on the higher-level stores and we drop
them after Persist, so that's what private MemCachedStore is for.

It doesn't improve things in any noticeable way, some ~1% can be observed in
neo-bench under various loads and even less than that in chain processing. But
it seems to be a bit better anyway (less allocations, less locks).
2022-02-17 22:27:39 +03:00
Roman Khimov
aefb26255a dao: drop DAO interface
It's a remnant from the days when we had Simple and Cached DAO
implementations, now it makes zero sense.
2022-02-16 18:24:20 +03:00
Roman Khimov
9d2ef775cf storage: simplify (*MemCachedStore).Put/Delete interface
They never return errors, so their interface should reflect that. This allows
to remove quite a lot of useless and never tested code.

Notice that Get still does return an error. It can be made not to do that, but
usually we need to differentiate between successful/unsuccessful accesses
anyway, so this doesn't help much.
2022-02-16 18:24:20 +03:00
Roman Khimov
be24bf6412 storage: drop Put and Delete from Store interface
It's only changed with PutChangeSet, single KV operations are handled by
MemCachedStore.
2022-02-16 18:24:20 +03:00
Roman Khimov
017795c9c1 storage: completely remove MemoryBatch
If you need something like that, just wrap another MemCachedStore layer around
it.
2022-02-16 16:13:12 +03:00
Roman Khimov
17a43b19e0 storage: remove Batch from Store
We never use it for real underlying stores, so these implementations are
useless (everything goes though PutChangeSet now).
2022-02-16 15:55:48 +03:00
Roman Khimov
35bdfc5eca storage: use two maps for MemoryStore
Simple and dumb as it is, this allows to separate contract storage from other
things and dramatically improve Seek() time over storage (even though it's
still unordered!) which in turn improves block processing speed.

        LevelDB             LevelDB (KeepOnlyLatest)  BoltDB              BoltDB (KeepOnlyLatest)
Master  real    16m27,936s  real    10m9,440s         real    16m39,369s  real    8m1,227s
        user    20m12,619s  user    26m13,925s        user    18m9,162s   user    18m5,846s
        sys     2m56,377s   sys     1m32,051s         sys     9m52,576s   sys     2m9,455s

2 maps  real    10m49,495s  real    8m53,342s         real    11m46,204s  real    5m56,043s
        user    14m19,922s  user    24m6,225s         user    13m25,691s  user    15m4,694s
        sys     1m53,021s   sys     1m23,006s         sys     4m31,735s   sys     2m8,714s

neo-bench performance is mostly unaffected, ~0.5% for 1-1 test and 4% for
10K-10K test both fall within regular test error range.
2022-02-16 15:55:48 +03:00
Roman Khimov
b6a4947cfd storage: drop STAccount prefix
It's unused for a long time.
2022-02-16 13:06:57 +03:00
Roman Khimov
54bc603831 core: remove old storage items synchronously during jump
There won't be a lot of them and GC experience we have shows that iterating
over 1-2M of entries is not a huge problem for DBs we have.
2022-02-16 13:03:13 +03:00
Roman Khimov
805d81e441 core: drop old transfer data during state jump 2022-02-16 12:12:03 +03:00
Roman Khimov
3a2b800e5f core: remove stale transfer entries, fix #2096
Initially I thought of doing it in the next persist cycle, but testing shows
that it needs just ~2-5% of the time MPT GC does, so doing it in the same
cycle doesn't affect anything.
2022-02-15 15:51:03 +03:00
Roman Khimov
c4b49a2d52 storage: deduplicate storage closing in tests 2022-02-14 17:29:21 +03:00
Roman Khimov
261552516b stateroot: use a better word for logs 2022-02-14 17:29:21 +03:00
Roman Khimov
ccdda21718 stateroot: add and use DataMPTAux for auxiliary data
Use DataMPT for nodes only, otherwise with 1M blocks with have 1M
height-stateroot mapping entries that our GC has to iterate over for no
reason.
2022-02-14 17:29:21 +03:00
Roman Khimov
ad606101c7 storage: add SeekGC interface for GC
It's very special, single-purpose thing, but it improves cumulative time spent
in GC by ~10% for LevelDB and by ~36% for BoltDB during 1050K mainnet chain
processing. While the overall chain import time doesn't change in any
noticeable way (~1%), I think it's still worth it, for machines with slower
disks the difference might be more noticeable.
2022-02-14 17:29:21 +03:00