Commit graph

1677 commits

Author SHA1 Message Date
Roman Khimov
33e37e60e5
Merge pull request #2264 from nspcc-dev/fix-win-tests
*: Windows compatibility fixes
2021-11-29 11:25:35 +03:00
AnnaShaleva
aefb6f9fee *: fix tests failing due to path.Join usage
Solution:
Use `file/filepath` package to construct expected path. This package is OS-aware, see https://github.com/golang/go/issues/30616.
2021-11-29 11:11:09 +03:00
AnnaShaleva
06beb4d534 core: fix TestMemCachedPersist test
Problem:
```
--- FAIL: TestMemCachedPersist (0.07s)
    --- FAIL: TestMemCachedPersist/BoltDBStore (0.07s)
        testing.go:894: TempDir RemoveAll cleanup: remove C:\Users\Anna\AppData\Local\Temp\TestMemCachedPersist_BoltDBStore294966711\001\test_bolt_db: The process cannot access the file because it is being used by another process.
```

Solution:
Release the resources occupied by the DB.
2021-11-26 18:26:27 +03:00
Evgeniy Stratonikov
fac595bbdf core: remove old storage items asynchronously
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:25:11 +03:00
Evgeniy Stratonikov
582d489c90 dao: add P2PStateExchangeExtensions setting to dao.Version
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:27 +03:00
Evgeniy Stratonikov
f1767f361d dao: add KeepOnlyLatestState setting to dao.Version
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:27 +03:00
Evgeniy Stratonikov
f7e2d3d717 dao: add stateroot-related settings to Version
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:27 +03:00
Evgeniy Stratonikov
856e9cf67b statesync: copy state by swapping prefix
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:27 +03:00
Evgeniy Stratonikov
6c5a7d9b29 dao: include settings in Version
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:26 +03:00
Evgeniy Stratonikov
856385b106 dao: use custom storage prefix
We use 2 prefixes for storing items because of state synchronization.
This commit allows to parametrize dao with the default prefix.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-24 16:22:07 +03:00
Roman Khimov
5fbe838fd4 core: add and use synchronous persist to avoid OOM
b9be892bf9 has made Persist asynchronous which
is very effective in allowing the system to continue processing
blocks/transactions while flushing things to disk. It at the same time is very
dangerous in that if the disk is slow and it takes much time to flush KV set
(more than persisting interval), there might be even bigger new KV set in
MemCachedStore by the time it finishes. Even if the system immediately starts
to flush this new data set it (being bigger) can take more time than the
previous one. And while doing so a new data set will appear in memory,
potentially again bigger than this.

So we can easily end up with the system going out of control, consuming more
and more memory and taking more and more time to persist a single set of
data. To avoid this we need to detect such condition and just wait for Persist
to really finish its job and release the resources.
2021-11-22 10:41:40 +03:00
Roman Khimov
aa06770b3d
Merge pull request #2266 from nspcc-dev/nep-11-tracking
NEP-11 tracking
2021-11-19 13:07:37 +03:00
Roman Khimov
ce9d0b22cf *: use NEP-XX naming consistently in docs/comments
Standards are NEP-11 and NEP-17, not NEP11, not NEP17, not anything
else. Variable/function names of course can use whatever fits, but documents
and comments should be consistent wrt this.
2021-11-19 12:58:46 +03:00
Roman Khimov
b622c1934d core: only save token logs if there is something to save
Makes no sense storing empty ones.
2021-11-18 00:09:10 +03:00
Roman Khimov
f3278ea5b2 core: reuse the transfer log buffer
Make it a bit more efficient.
2021-11-18 00:09:10 +03:00
Roman Khimov
125e4231b0 core: store NEP-11 transfers, add accessor functions 2021-11-18 00:09:10 +03:00
Roman Khimov
7f40a0cfd8 state: prime token transfer info with the demand for new logs
Setting NewNEPXXBatch avoids the need to get them from the database which is
useful for newly tracked accounts.
2021-11-16 22:52:10 +03:00
Roman Khimov
095ed3f64e dao: drop unused AppendNEP17Transfer() 2021-11-16 22:51:04 +03:00
Roman Khimov
c63aeb38bb state: prepare for NEP-11 data tracking, refactor/rename
There is a lot of similarity, so try reusing common code and use more neutral
naming.
2021-11-16 19:18:06 +03:00
Roman Khimov
ecfe3e7247
Merge pull request #2260 from nspcc-dev/fix-notary-test-failure
Fix notary test failure
2021-11-16 16:09:41 +03:00
Roman Khimov
3b08b5b19b native: add NEP-11 tracking to management contract 2021-11-16 13:28:59 +03:00
Roman Khimov
f96d9d168e native/interop: use oracle request signers for oracle response witness
Oracle responses must use the same set of signers as oracle requests even
though the transaction itself is signed by oracle nodes/contract.

We can probably improve interop.Context by removing Tx field completely and
adding more functionality to Container, but it's not very convenient for
VerifyWitness and will require adding more stub-like methods for Block, so Tx
is used for now (and we do have it in every relevant case).
2021-11-16 11:51:15 +03:00
Roman Khimov
65016e8070 core: fix data race in notary test
I don't think it's possible with regular service functioning, but it happens
during testing because of pointer reuse:

WARNING: DATA RACE
Read at 0x00c003a0e3f0 by goroutine 114:
  github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).verifyIncompleteWitnesses()
      /home/runner/work/neo-go/neo-go/pkg/services/notary/notary.go:441 +0x1dc
  github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).OnNewRequest()
      /home/runner/work/neo-go/neo-go/pkg/services/notary/notary.go:188 +0x205
  github.com/nspcc-dev/neo-go/pkg/core.TestNotary.func11()
      /home/runner/work/neo-go/neo-go/pkg/core/notary_test.go:347 +0x612
  github.com/nspcc-dev/neo-go/pkg/core.TestNotary()
      /home/runner/work/neo-go/neo-go/pkg/core/notary_test.go:443 +0xe33
  testing.tRunner()
      /opt/hostedtoolcache/go/1.16.10/x64/src/testing/testing.go:1193 +0x202

Previous write at 0x00c003a0e3f0 by goroutine 104:
  github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).finalize()
      /home/runner/work/neo-go/neo-go/pkg/services/notary/notary.go:338 +0x50a
  github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).PostPersist()
      /home/runner/work/neo-go/neo-go/pkg/services/notary/notary.go:314 +0x297
  github.com/nspcc-dev/neo-go/pkg/services/notary.(*Notary).Run()
      /home/runner/work/neo-go/neo-go/pkg/services/notary/notary.go:169 +0x4a7
2021-11-15 17:33:02 +03:00
Roman Khimov
f48dffe0e6 core: fix fallback VUB value in notary test
Serializing/deserializing the payload yields this:

 Error:          Received unexpected error:
                 both main and fallback transactions should have the same ValidUntil value
2021-11-15 17:32:14 +03:00
Roman Khimov
b31a8d750e
Merge pull request #2252 from nspcc-dev/packmap-packstruct
vm: add PACKMAP/PACKSTRUCT, extend UNPACK
2021-11-12 18:25:54 +03:00
Roman Khimov
9cdd360606
Merge pull request #2251 from nspcc-dev/rules-witness-scope
transaction: add new Rules witness scope
2021-11-12 18:25:39 +03:00
Roman Khimov
3e6ce3c221 vm: add PACKMAP/PACKSTRUCT, extend UNPACK
See neo-project/neo-vm#443.
2021-11-12 17:19:59 +03:00
Roman Khimov
9875799893 transaction: add new Rules witness scope
See neo-project/neo#2622. The implementation is somewhat asymmetric (and not
very efficient) for binary/JSON encoding/decoding, but it should be
sufficient.
2021-11-12 15:29:28 +03:00
Roman Khimov
0a7f8afcea
Merge pull request #2229 from nspcc-dev/contract-test
Generic contract testing infrastructure
2021-11-11 11:46:55 +03:00
Evgeniy Stratonikov
ce549e4cb2 core/test: allow to determine system fee automatically
Eventually this will be replaced by `pkg/neotest` invocations but for
now it allows us to remove NNS constants together with the tests.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-11 11:15:15 +03:00
Evgeniy Stratonikov
e3625152c6 core: move NNS test out of core
Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-11-11 11:15:15 +03:00
Roman Khimov
6b8e615094
Merge pull request #2234 from nspcc-dev/rpc/params-parsing
rpc: method-specific parameters parsing optimisation
2021-11-10 20:45:44 +03:00
Anna Shaleva
1a32fcb2dc core: specify method not found call error
It's useful for debugging and external users.
2021-11-08 17:19:00 +03:00
Roman Khimov
328f8b7954
Merge pull request #2238 from nspcc-dev/fix-block-queue
network: remove priority queue from block queue
2021-11-01 12:03:45 +03:00
Roman Khimov
8bb1ecb45a network: remove priority queue from block queue
Use circular buffer which is a bit more appropriate. The problem is that
priority queue accepts and stores equal items which wastes memory even in
normal usage scenario, but it's especially dangerous if the node is stuck for
some reason. In this case it'll accept from peers and put into queue the same
blocks again and again leaking memory up to OOM condition.

Notice that queue length calculation might be wrong in case circular buffer
wraps, but it's not very likely to happen (usually blocks not coming from the
queue are added by consensus and it's not very fast in doing so).
2021-11-01 11:49:01 +03:00
Roman Khimov
67eac3a27f
Merge pull request #2196 from nspcc-dev/drop-badger-and-redis
Drop Badger and Redis
2021-10-28 16:08:49 +03:00
AnnaShaleva
3fed7b8eec core: handle empty MPT batch properly
It's OK to have it.
2021-10-27 18:44:39 +03:00
Roman Khimov
1144a03486 storage: drop RedisDB, close #2130 2021-10-27 17:32:25 +03:00
Roman Khimov
fb4b87bb96 storage: drop BadgerDB support, close #2130 2021-10-27 17:31:55 +03:00
AnnaShaleva
2d196b3f35 rpc: refactor calculatenetworkfee handler
Use (Blockchainer).VerifyWitness() to calculate network fee for
contract-based witnesses.
2021-10-25 19:07:25 +03:00
Anna Shaleva
75d7891ca1 services: add ability to combine notary signers
Notes for witnesses:
* [N sig + M multisig + K contract] combination is possible where N, M, K >=0.
* Each verification script should be properly filled in.
* Each invocation script should either be empty or contain exactly one
signature.
2021-10-25 12:22:13 +03:00
Roman Khimov
d551439654
Merge pull request #2193 from nspcc-dev/optimize-find
core: optimise (*MemCachedStorage).Seek
2021-10-21 21:20:33 +03:00
Anna Shaleva
3450371910 core: split (*MemCachedStore) Seek and SeekAsync methods
Use SeekAsync for System.Storage.Find and Seek for the rest of cases.
2021-10-21 10:05:12 +03:00
Anna Shaleva
dcda7bec63 core: squash PS-seeking and merging routines in (*MemCachedStore).Seek
We don't need a separate routine to merge seek results.
2021-10-21 10:05:12 +03:00
Anna Shaleva
c88720bf45 core: remove memstore routine from (*MemCachedStore).SeekAsync
It adds unnecessary overhead to computations.
2021-10-21 10:05:12 +03:00
Anna Shaleva
dfe2c667e1 core: do not hold the lock while seeking over persistent store 2021-10-21 10:05:12 +03:00
Anna Shaleva
07cbe4d253 core: add finalizer functions to interop context
These functions are aimed to free the resources occupied by storage
iterator by the end of script execution or whenever Finilize is called.
2021-10-21 10:05:12 +03:00
Anna Shaleva
0a4f45c9b0 core: add ability to free storage.Iterator resources 2021-10-21 10:05:12 +03:00
Anna Shaleva
89ee2e7720 core: refactor storage.Find and storage.Iterator to work with channel
Add SeekAsync methods in order to fetch matching storage items
on demand. Refactor storage.Find and storage.Iterator wrt these changes.
2021-10-21 10:05:12 +03:00
Anna Shaleva
f2ac07a3c0 core: do not copy key/value results got from MemCachedStore.Seek
They are already copied inside the MemCachedStore.Seek, so that
persistent storage  can't change them anymore.
2021-10-21 10:05:12 +03:00