Commit graph

5198 commits

Author SHA1 Message Date
Roman Khimov
3eed9d06f8 stackitem: add some hint to 'seen' maps
It's empirical, we usually have one container, but four is likely to fit most
of regular cases.
2021-12-01 21:36:25 +03:00
Roman Khimov
9576e10d04 storage: use map size hints to optimize subsequent persist()
We're likely to have something comparable to the current changeset in the
subsequent one. If it's bigger, no big deal, it'll be reallocated, if it's
smaller, no big deal, the next one will be preallocated smaller.
2021-12-01 21:36:25 +03:00
Roman Khimov
428ba48444 mempool: use capacity parameter to avoid map reallocations
Which is what happens in AddBlock that fills a new pool with exactly the
number of transactions we know already.
2021-12-01 21:36:25 +03:00
Roman Khimov
e236ac4067 io: optimize WriteString, avoid allocation
name                          old time/op    new time/op    delta
AppExecResult_EncodeBinary-8     477ns ± 0%     467ns ± 3%     ~     (p=0.190 n=4+5)

name                          old alloc/op   new alloc/op   delta
AppExecResult_EncodeBinary-8      376B ± 0%      368B ± 0%   -2.13%  (p=0.008 n=5+5)

name                          old allocs/op  new allocs/op  delta
AppExecResult_EncodeBinary-8      5.00 ± 0%      4.00 ± 0%  -20.00%  (p=0.008 n=5+5)
2021-12-01 21:36:25 +03:00
Roman Khimov
748a70569a mempool: use uint256 library for fee tracking
It's very effective in avoiding allocations for big.Int, we don't have a
microbenchmark for memppol, but this improves TPS metrics by ~1-2%, so it's
noticeable.
2021-12-01 21:36:25 +03:00
Roman Khimov
ee05f73b6f vm: simplify slot and make it private
Hiding refcounter inside a slot is actually a good idea, but it makes the
structure somewhat bigger, especially given that the refcounter is the same
and belongs more to VM or Context. New structure is a bit more efficient:

name                    old time/op    new time/op    delta
ScriptFibonacci-8          672µs ± 2%     644µs ± 0%  -4.15%  (p=0.008 n=5+5)
ScriptNestedRefCount-8    1.08ms ± 1%    1.05ms ± 2%  -2.56%  (p=0.008 n=5+5)
ScriptPushPop/4-8         1.52µs ± 1%    1.47µs ± 1%  -3.14%  (p=0.008 n=5+5)
ScriptPushPop/16-8        3.66µs ± 1%    3.54µs ± 1%  -3.24%  (p=0.008 n=5+5)
ScriptPushPop/128-8       24.7µs ± 1%    23.2µs ± 1%  -6.14%  (p=0.008 n=5+5)
ScriptPushPop/1024-8       183µs ± 1%     173µs ± 1%  -5.01%  (p=0.008 n=5+5)

name                    old alloc/op   new alloc/op   delta
ScriptFibonacci-8          114kB ± 0%     114kB ± 0%    ~     (p=0.079 n=4+5)
ScriptNestedRefCount-8     241kB ± 0%     241kB ± 0%    ~     (p=0.333 n=5+4)
ScriptPushPop/4-8           160B ± 0%      160B ± 0%    ~     (all equal)
ScriptPushPop/16-8          640B ± 0%      640B ± 0%    ~     (all equal)
ScriptPushPop/128-8       8.70kB ± 0%    8.70kB ± 0%    ~     (all equal)
ScriptPushPop/1024-8      73.2kB ± 0%    73.2kB ± 0%    ~     (all equal)

name                    old allocs/op  new allocs/op  delta
ScriptFibonacci-8          3.17k ± 0%     3.17k ± 0%  -0.03%  (p=0.008 n=5+5)
ScriptNestedRefCount-8     10.7k ± 0%     10.7k ± 0%    ~     (all equal)
ScriptPushPop/4-8           8.00 ± 0%      8.00 ± 0%    ~     (all equal)
ScriptPushPop/16-8          32.0 ± 0%      32.0 ± 0%    ~     (all equal)
ScriptPushPop/128-8          259 ± 0%       259 ± 0%    ~     (all equal)
ScriptPushPop/1024-8       2.05k ± 0%     2.05k ± 0%    ~     (all equal)

It'd be especially nice to internalize static slot, but as we can't compare
slices it's not possible.
2021-12-01 21:36:25 +03:00
Roman Khimov
a0473aca92
Merge pull request #2283 from nspcc-dev/fix-vm-tests
vm: fix cli tests
2021-12-01 11:35:41 +03:00
Roman Khimov
01d15ff473
Merge pull request #2270 from nspcc-dev/vm-invoked-contracts
Add invoked contract tracing
2021-12-01 11:27:27 +03:00
AnnaShaleva
c67d249a8d vm: fix cli tests
* Remove extra symbols during command execution.
* Fix test contract paths.
2021-12-01 11:14:29 +03:00
Roman Khimov
c663d3f39f
Merge pull request #2279 from nspcc-dev/minor-fixes
services: fix Oracle response creation algorithm
2021-11-30 18:15:50 +03:00
Roman Khimov
dbe184a335
Merge pull request #2280 from nspcc-dev/compiler-struct-cast
compiler: allow to slice struct field
2021-11-30 18:14:56 +03:00
Roman Khimov
c14376473b
Merge pull request #2282 from nspcc-dev/defer-after-interop
Fix defers in functions using inlined calls
2021-11-30 18:14:13 +03:00
Roman Khimov
f12f871432 compiler: use new defer stack for inlined functions, fix #2281
Inlined functions shouldn't care about outer scope defer statements, they
can't.
2021-11-30 18:04:02 +03:00
Roman Khimov
cddd5e1f6d compiler: add a panic on internal lambda inconsistency
It's a bit better than behavior observer in #2281 (which is very cryptic).
2021-11-30 18:02:38 +03:00
Evgeniy Stratonikov
66a4245bef compiler: allow to slice struct field
It makes no sense to restrict to identifiers.
2021-11-30 15:48:46 +03:00
Anna Shaleva
1d1538c566 services: fix Oracle response transaction creation
Problem: transactions with wrong hashes are accepted to the chain if
consensus nodes are designated as Oracle nodes. The result is wrong
MerkleRoot for the accepted block. Consensus nodes got such blocks
right from the dbft and store them without errors, but if
non-consensus nodes are present in the network, they just can't accept
these "bad" blocks:
```
2021-11-29T12:56:40.533+0300	WARN	blockQueue: failed adding block into the blockchain	{"error": "invalid block: MerkleRoot mismatch (expected a866b57ad637934f7a7700e3635a549387e644970b42681d865a54c3b3a46122, calculated d465aafabaf4539a3f619d373d178eeeeab7acb9847e746e398706c8c1582bf8)", "blockHeight": 17, "nextIndex": 18}
```

This problem happens because of transaction hash caching. We can't set
transaction hash if transaction construction wasn't yet completed.
2021-11-30 10:43:58 +03:00
Anna Shaleva
6101d4b251 cli: improve error messages for signStoredTransaction
It's nice for user when he sees meaningful error messages.
2021-11-29 11:34:07 +03:00
Roman Khimov
33e37e60e5
Merge pull request #2264 from nspcc-dev/fix-win-tests
*: Windows compatibility fixes
2021-11-29 11:25:35 +03:00
AnnaShaleva
5ef9f4498e .github: add job to publish image for WSC
Also, remove Go setup from publishing jobs because we don't need Go on runner.
2021-11-29 11:13:54 +03:00
AnnaShaleva
d2d75b8ba5 .github: unify Tests workflow 2021-11-29 11:13:49 +03:00
AnnaShaleva
e64a617844 .github: add action to build WSC image 2021-11-29 11:11:09 +03:00
AnnaShaleva
b940167a6b .github: create separate workflow to build CLI/image
There's too much jobs in Tests workflow, so we can split them into several parts.
2021-11-29 11:11:09 +03:00
AnnaShaleva
f97f6168fc .github: remove golang from build_image job
We're building inside Docker container, so we don't need golang to
be installed on the host machine.
2021-11-29 11:11:09 +03:00
AnnaShaleva
7770535d4f .github: add job to build binaries on Windows 2021-11-29 11:11:09 +03:00
AnnaShaleva
ebd128ee79 .github: add windows-based tests workflow 2021-11-29 11:11:09 +03:00
AnnaShaleva
16da133d61 cli: release resources occupied by chain before dumpDB exit
Problem:
```
--- FAIL: TestDumpDB (0.08s)
--- FAIL: TestDumpDB/too_low_chain
testing.go:894: TempDir RemoveAll cleanup: remove C:\Users\Anna\AppData\Local\Temp\TestDumpDB_too_low_chain357310492\001\chains\privnet\000001.log: The process cannot access the file because it is being used by another process.
```

Solution:
Release resources occupied by the chain even on non-error command exit.
2021-11-29 11:11:09 +03:00
AnnaShaleva
aefb6f9fee *: fix tests failing due to path.Join usage
Solution:
Use `file/filepath` package to construct expected path. This package is OS-aware, see https://github.com/golang/go/issues/30616.
2021-11-29 11:11:09 +03:00
Anna Shaleva
6cdb701a9d vm: improve error message for (*executor).checkError
It is helpful if something goes wrong.
2021-11-29 11:09:39 +03:00
AnnaShaleva
7c48177bf7 io: fix TestMakeDirForFile* tests
Problem:
```
--- FAIL: TestMakeDirForFile_HappyPath (0.01s)
    testing.go:894: TempDir RemoveAll cleanup: remove C:\Users\Anna\AppData\Local\Temp\TestMakeDirForFile_HappyPath402638411\001\testDir\testFile.test: The process cannot access the file because it is being used by another process.
--- FAIL: TestMakeDirForFile_Negative (0.01s)
    testing.go:894: TempDir RemoveAll cleanup: remove C:\Users\Anna\AppData\Local\Temp\TestMakeDirForFile_Negative672737582\001\testFile.test: The process cannot access the file because it is being used by another process.
FAIL
```

Solution:
Release resources occupied by os.Create.
2021-11-26 18:26:27 +03:00
AnnaShaleva
06beb4d534 core: fix TestMemCachedPersist test
Problem:
```
--- FAIL: TestMemCachedPersist (0.07s)
    --- FAIL: TestMemCachedPersist/BoltDBStore (0.07s)
        testing.go:894: TempDir RemoveAll cleanup: remove C:\Users\Anna\AppData\Local\Temp\TestMemCachedPersist_BoltDBStore294966711\001\test_bolt_db: The process cannot access the file because it is being used by another process.
```

Solution:
Release the resources occupied by the DB.
2021-11-26 18:26:27 +03:00
AnnaShaleva
991f1be8e6 cli: fix TestHandleLoggingParams test
Problem:
```
--- FAIL: TestHandleLoggingParams (0.02s)
    --- FAIL: TestHandleLoggingParams/default (0.00s)
        server_test.go:51:
                Error Trace:    server_test.go:51
                Error:          Received unexpected error:
                                couldn't open sink "C:\\Users\\Anna\\AppData\\Local\\Temp\\TestHandleLoggingParams226652490\\001/file.log": no sink found for scheme "c"
                Test:           TestHandleLoggingParams/default
    --- FAIL: TestHandleLoggingParams/debug (0.00s)
        server_test.go:64:
                Error Trace:    server_test.go:64
                Error:          Received unexpected error:
                                couldn't open sink "C:\\Users\\Anna\\AppData\\Local\\Temp\\TestHandleLoggingParams226652490\\001/file.log": no sink found for scheme "c"
                Test:           TestHandleLoggingParams/debug
```

Solution:
Currently no solution is implemented, but we can use relative paths instead of absolute.
2021-11-26 18:26:20 +03:00
Roman Khimov
14cb3181ab
Merge pull request #2278 from nspcc-dev/config/fix-oracle
config: use http for oracle node addresses
2021-11-26 17:31:36 +03:00
Anna Shaleva
e2ef904901 config: use http for oracle nodes
Otherwise the following error occurs:
```
first path segment in URL cannot contain colon
```
2021-11-26 17:17:10 +03:00
Roman Khimov
b455e9338c
Merge pull request #2275 from nspcc-dev/add-more-oracle-logs
oracle: unify filtering and add log for error case
2021-11-25 09:51:48 +03:00
Roman Khimov
550372a790
Merge pull request #2201 from nspcc-dev/mpt-state-switch
Faster state switch
2021-11-25 09:51:11 +03:00
Evgeniy Stratonikov
fac595bbdf core: remove old storage items asynchronously
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:25:11 +03:00
Evgeniy Stratonikov
582d489c90 dao: add P2PStateExchangeExtensions setting to dao.Version
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:27 +03:00
Evgeniy Stratonikov
f1767f361d dao: add KeepOnlyLatestState setting to dao.Version
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:27 +03:00
Evgeniy Stratonikov
f7e2d3d717 dao: add stateroot-related settings to Version
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:27 +03:00
Evgeniy Stratonikov
856e9cf67b statesync: copy state by swapping prefix
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:27 +03:00
Evgeniy Stratonikov
6c5a7d9b29 dao: include settings in Version
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:26 +03:00
Evgeniy Stratonikov
856385b106 dao: use custom storage prefix
We use 2 prefixes for storing items because of state synchronization.
This commit allows to parametrize dao with the default prefix.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:07 +03:00
Roman Khimov
6de6415a53 oracle: unify filtering and add log for error case
Just to simplify debugging.
2021-11-23 18:43:15 +03:00
Roman Khimov
019af7de46
Merge pull request #2271 from nspcc-dev/fix-sync-oom
core: add and use synchronous persist to avoid OOM
2021-11-23 12:37:48 +03:00
Roman Khimov
a6e5c18359
Merge pull request #2272 from nspcc-dev/fix-sender-custom-scopes
Fix custom sender scopes in CreateTxFromScript
2021-11-23 12:37:33 +03:00
Roman Khimov
e5b5a3b118 cli: add scope check tests for invocations
It doesn't trigger the problem fixed in the previous commit because here
CreateTxFromScript is used with pre-calculated GAS value from InvokeFunction
and thus it's not performing invocation. Still, it's a nice test extension.
2021-11-22 12:46:42 +03:00
Roman Khimov
f58d424c6d rpc/client: fix custom* scope processing in CreateTxFromScript
Copying just the scope doesn't work for CustomContracts, CustomGroups and
Rules because they all contain additional metadata. Thanks @mialbu for
reporting this.
2021-11-22 12:44:42 +03:00
Roman Khimov
5fbe838fd4 core: add and use synchronous persist to avoid OOM
b9be892bf9 has made Persist asynchronous which
is very effective in allowing the system to continue processing
blocks/transactions while flushing things to disk. It at the same time is very
dangerous in that if the disk is slow and it takes much time to flush KV set
(more than persisting interval), there might be even bigger new KV set in
MemCachedStore by the time it finishes. Even if the system immediately starts
to flush this new data set it (being bigger) can take more time than the
previous one. And while doing so a new data set will appear in memory,
potentially again bigger than this.

So we can easily end up with the system going out of control, consuming more
and more memory and taking more and more time to persist a single set of
data. To avoid this we need to detect such condition and just wait for Persist
to really finish its job and release the resources.
2021-11-22 10:41:40 +03:00
Roman Khimov
c01427ca65 rpc: add verbose parameter to invokefunction and invokescript 2021-11-20 21:57:41 +03:00
Roman Khimov
5b470f14cc rpc: pass single Param to CreateFunctionInvocationScript
N3 contracts require method and array of arguments to function correctly, but
CreateFunctionInvocationScript can produce scripts that won't work correctly
if one is to pass anything but a single array parameter to it. This doesn't
make much sense, so we can always expect there to be an array of parameters in
the third positional invokefunction argument (the same way C# node does).
2021-11-20 21:57:41 +03:00