Commit graph

137 commits

Author SHA1 Message Date
Roman Khimov
13bf2618ef storage: improve PutBatch for MemoryStore
Make it look more like a real transaction, put/delete things with a single
lock. Make a copy of value in Put also, just for safety purposes, no one knows
how this value slice can be used after the Put.
2019-10-15 12:56:25 +03:00
Roman Khimov
3ada92944a storage: drop Len from the Batch interface
It's almost meaningless now and we can easily live without it.
2019-10-15 12:56:25 +03:00
Roman Khimov
e111892653 storage: redo DB testing
Make generic tests for all Store implementations, deduplicate tests. Implement
Delete() tests, test Seek() better, add LevelDB tests (finally!).
2019-10-15 12:56:25 +03:00
Roman Khimov
48b6a427cf storage: add Delete method for Batch and Store
It's gonna be used by Storage and Contract interops, both can delete their
data.
2019-10-07 19:40:11 +03:00
Roman Khimov
add9368e9d storage: use strings as keys for memory batch
Using pointers is just plain wrong here, because the batch can be updated with
newer values for the same keys.

Fixes Seek() to use HasPrefix also because this is the intended behavior.
2019-10-07 17:05:53 +03:00
Roman Khimov
7779ad6c28 storage: always return ErrKeyNotFound when key is not found
It was true for MemoryStore and BoltDB, but not for LevelDB and Redis.
2019-09-27 15:42:36 +03:00
Roman Khimov
5ff1399d54 storage: unexport MemoryStore mutex
It shouldn't be available from outside.
2019-09-27 15:42:36 +03:00
Roman Khimov
0dbdbb9c2a storage: remove RedisBatch in favor of MemoryBatch
They're also almost the same.
2019-09-27 15:42:35 +03:00
Roman Khimov
af557cea19 storage: deduplicate BoltDBBatch/MemoryBatch
BoltDB doesn't have internal batching mechanism, thus we have a substitute for
it, but this substitute is absolutely identical to MemoryBatch, so it's better
to unify them and import ac5d2f94d3 fix into the
MemoryBatch.
2019-09-27 15:42:35 +03:00
Roman Khimov
8bf37f45fc storage: remove impossible error branch from MemoryStore
MemoryStore is always successful, so this branch makes no sense.
2019-09-27 15:42:35 +03:00
Roman Khimov
4a732d97ac storage: add comments about errors from MemoryStore
It never returns errors for some operations and this knowledge can be used by
other code.
2019-09-27 15:42:35 +03:00
Roman Khimov
920d7c610c core: remove blockCache, use MemoryStore, redesign persist()
Commit 578ac414d4 was wrong in that it saved
only a part of the block, so depending on how you use blockchain, you may
still see that the block was not really processed properly. To really fix it
this commit introduces intermediate storage layer in form of memStore, which
actually is a MemoryStore that supports full Store API (thus easily fitting
into the existing code) and one extension that allows it to flush its data to
some other Store.

It also changes AddBlock() semantics in that it only accepts now successive
blocks, but when it does it guarantees that they're properly added into the
Blockchain and can be referred to in any way. Pending block queing is now
moved into the server (see 8c0c055ac657813fe3ed10257bce199e9527d5ed).

So the only thing done with persist() now is just a move from memStore to
Store which probably should've always been the case (notice also that
previously headers and some other metadata was written into the Store
bypassing caching/batching mechanism thus leading to some inefficiency).
2019-09-27 15:42:35 +03:00
Roman Khimov
ac5d2f94d3 storage: fix BoltDB batched Put()
It must copy both the value and the key because they can be reused for other
purposes between Put() and PutBatch(). This actually happens with values in
headers processing, leading to wrong data being written into the DB.

Extend the batch test to check for that.
2019-09-23 09:27:18 +03:00
Roman Khimov
5cddfe071b
Merge pull request #403 from nspcc-dev/fix_rpctest
Fix rpctest, fix #353, fix #305.
2019-09-18 23:14:37 +03:00
Vsevolod Brekelov
55dfc0bbbc storage: add seek implementation
during testing found missing Seek() implementation for inmemorydb
2019-09-18 18:20:41 +03:00
Roman Khimov
cb4be2ae4a storage: drop useless README, refs. #339 2019-09-17 17:30:05 +03:00
Roman Khimov
5bf00db2c9 io: move BinReader/BinWriter there, redo Serializable with it
The logic here is that we'll have all binary encoding/decoding done via our io
package, which simplifies error handling. This functionality doesn't belong to
util, so it's moved.

This also expands BufBinWriter with Reset() method to fit the needs of core
package.
2019-09-16 23:39:51 +03:00
Roman Khimov
aacf58c9ab util: add 'constructors' for BinReader/BinWriter
And an additional BufBinWriter to ease buffer management.
2019-09-16 23:38:48 +03:00
Vsevolod Brekelov
264dfef370 storage: close function
add close function to storage interface
add common defer function call which will close db connection
remove context as soon as it's not needed anymore
updated unit tests
2019-09-16 18:52:47 +03:00
Vsevolod Brekelov
2cabd5b274 storage: add boltDB unit tests 2019-09-14 10:29:08 +03:00
Vsevolod Brekelov
bfe5fd538e storage: add boltDB support
closes #335
2019-09-14 10:28:43 +03:00
Roman Khimov
4395cea344 storage: fix reading overlapping hash blocks from the DB
In the unlikely event of overlapping hash block written to the DB we might end
up with wrong hash list. That happened to me for some reason when synching
with the testnet leading to the following keys with respective values:
  150000 -> 2000 hashes
  152000 -> 2000 hashes
  153999 -> 2000 hashes

Reading it hashes number 153999 and 154000 got the same values and the chain
couldn't sync correctly.
2019-09-11 20:33:41 +03:00
Vsevolod Brekelov
06ecb1855e storage: redis db unit test 2019-09-10 17:22:45 +03:00
Vsevolod Brekelov
4f680703a4 storage: multiple DB support and Redis support
add config which closes #336
add redis db support
2019-09-10 17:22:21 +03:00
Roman Khimov
361724a33e pkg/core: use util.binaryReader/Writer
Simplify error handling.
2019-08-28 19:45:17 +03:00
Roman Khimov
07096a551b util: move strange Read2000Uint256Hashes() into storage
It's the only user of it.
2019-08-28 14:38:57 +03:00
Evgeniy Kulikov
33f59debf2
store: Fix DataRace in MemoryStore
- Add RWMutex to MemoryStore struct
- Use Lock/Unlock for writing
- Use RLock/RUnlock for reading
- Fix #313 issue
2019-08-27 16:48:30 +03:00
Roman Khimov
d5d570f793 uint256: add Reverse(), change String() to be BE
This one makes a little more obvious that we're operating with LE
representations mostly. Refs. #307. See #314 also.
2019-08-26 13:32:19 +03:00
Evgeniy Kulikov
cdba88b9f2 Code refactoring (#143)
- simplify code
- prealoc slices
- check errors in tests
- regexp Compile replaced with MustCompile
- uint* cannot be negative
2019-02-19 14:22:33 +01:00
Evgeniy Kulikov
9c24bf9139 Blockchain graceful shutdown (#139)
* Blockchain graceful shutdown

- fix #138
- stop blockchain goroutine
- close leveldb database

* fix possible context leak (go vet)
2019-02-19 12:48:48 +01:00
Evgeniy Kulikov
67cbcac643 Fix typos (#133)
* Fix typos

* revert chains/unit_testnet

* revert chains

* fix review comments (thx @AlexVanin)
2019-02-13 18:01:10 +00:00
Evgeniy Kulikov
630919bf7d Fix typos and warnings for GoReport / GolangCiLinter (#132)
- typos
- gofmt -s
- govet warnings
- golangci-lint run
2019-02-09 16:53:58 +01:00
Evgeniy Kulikov
f000b76879 [FIX] Formatting and code-style (#118)
* [FIX] Formatting and code-style

- gofmt
- import resort
- prealloc slices
- simplify code

* fix vet
2019-01-25 12:20:35 +01:00
Evgeniy Kulikov
6ccb518ab0 Optimizations + some improvements (#105)
* Optimizations + some improvements

- optimized pkg/core/storage.HeaderHashes
- optimized pkg/rpc.performRequest (used json.Encoder)
- fixes for pkg/util.ReadVarUint and pkg/util.WriteVarUint
- optimized and fix fixed8 (Fixed8DecodeString / MarshalJSON) + tests
- optimized and fix uint160 (Bytes / Uint160DecodeString / Equal / MarshalJSON) + tests
- optimized and fix uint256 (Bytes / Equal / MarshalJSON) + tests
- preallocate for pkg/vm.buildStackOutput
- add go.mod / go.sum

* update version
2018-11-26 16:56:45 +01:00
Anthony De Meulemeester
b2021c126e
Tweaks for network and storage (#66)
* Made Encode/Decode message public.

* Added Redis storage driver and made some optimizations for the initialising the blockchain

* removed log lines in tcp_peer

* Added missing comments on exported methods.

* bumped version
2018-04-09 18:58:09 +02:00
Anthony De Meulemeester
52fa41a12a
Persist transactions (#51)
* added account_state + changed ECPoint to PublicKey

* account state persist

* in depth test for existing accounts.

* implemented GetTransaction.

* added enrollment TX

* added persist of accounts and unspent coins

* bumped version -> 0.32.0
2018-03-21 17:11:04 +01:00
Anthony De Meulemeester
a67728628e
Persist blockchain with leveldb on disk (#48)
* Created test_data folder with block json files for testing + create separate file for block base.

* Fixed bug in WriteVarUint + Trim logic + unit tests

* Refactored store and add more tests for it.

* restore headerList from chain file

* Fix tx decode bug + lots of housekeeping.

* Implemented Node restore state from chain file.

* Created standalone package for storage. Added couple more methods to Batch and Store interfaces.

* Block persisting + tests

* bumped version -> 0.31.0
2018-03-17 12:53:21 +01:00