Commit graph

72 commits

Author SHA1 Message Date
Roman Khimov
d9ee31fb52 *: use clear() to clear maps
Supposedly more efficient since we can avoid some memory management dances.

Memory pool agrees:

goos: linux
goarch: amd64
pkg: github.com/nspcc-dev/neo-go/pkg/core/mempool
cpu: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics
                       │  pool.old   │              pool.new               │
                       │   sec/op    │   sec/op     vs base                │
Pool/one,_incr_fee-16    12.44m ± 1%   12.51m ± 1%   +0.55% (p=0.029 n=10)
Pool/many,_same_fee-16   4.960m ± 2%   3.100m ± 1%  -37.50% (p=0.000 n=10)
Pool/many,_incr_fee-16   16.03m ± 2%   14.11m ± 1%  -12.00% (p=0.000 n=10)
Pool/one,_same_fee-16                  1.742m ± 1%
geomean                  9.964m        5.556m       -17.92%

                       │    pool.old     │                pool.new                │
                       │      B/op       │      B/op       vs base                │
Pool/one,_incr_fee-16     8.117Ki ± 120%   7.101Ki ± 128%  -12.52% (p=0.022 n=10)
Pool/many,_same_fee-16   3941.2Ki ±   0%   805.4Ki ±   0%  -79.56% (p=0.000 n=10)
Pool/many,_incr_fee-16   3936.2Ki ±   0%   829.8Ki ±   0%  -78.92% (p=0.000 n=10)
Pool/one,_same_fee-16                      12.98Ki ±  10%
geomean                   501.2Ki          88.59Ki         -66.47%

                       │   pool.old   │               pool.new               │
                       │  allocs/op   │  allocs/op    vs base                │
Pool/one,_incr_fee-16     28.00 ± 21%    24.00 ± 21%  -14.29% (p=0.002 n=10)
Pool/many,_same_fee-16   40.38k ±  0%   40.03k ±  0%   -0.86% (p=0.000 n=10)
Pool/many,_incr_fee-16   40.38k ±  0%   40.04k ±  0%   -0.85% (p=0.000 n=10)
Pool/one,_same_fee-16                    23.00 ±  4%
geomean                  3.574k          969.8         -5.55%

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2024-08-24 22:41:48 +03:00
Roman Khimov
5d1d7b104e mempool: properly remove fees when removing tx during Add
Fixes #3488.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2024-08-05 15:36:14 +03:00
Roman Khimov
a11e433754 mempool: move metrics out of removeInternal, simplify it
Metrics should be updated once per action, currently removeInternal is
used by Add and Remove, the first one updates them in the end anyway and
remove should do the same.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2024-07-30 17:43:14 +03:00
Roman Khimov
6334192a95 mempool: remove Feer from Remove()
It's not used and not needed for removal.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2024-07-30 17:37:18 +03:00
Anna Shaleva
dc3d1300dd *: replace go.uber.org/atomic with sync/atomic
Use sync/atomic everywhere and exclude go.uber.org/atomic from go.mod.
Close #2626.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
2023-10-12 17:59:34 +03:00
Anna Shaleva
f3c1283ac6 *: move NVB and Conflicts attributes out of extensions
They're a part of the regular protocol now.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2023-09-04 16:39:44 +03:00
Tatiana Nesterenko
d285342d54 core: add function IterateVerifiedTransactions
IterateVerifiedTransactions iterates through verified transactions in
memory pool and invokes function cont. Where cont callback returns
whether we should continue with the traversal process.

Signed-off-by: Tatiana Nesterenko <tatiana@nspcc.io>
2023-08-31 18:51:43 +01:00
Anna Shaleva
ee4b8f883b core: check signers of on-chained conflict during new tx verification
During new transaction verification if there's an on-chain conflicting
transaction, we should check the signers of this conflicting transaction.
If the signers intersect with signers of the incoming transaction, then
the conflict is treated as valid and verification for new incoming
transaction should fail. Otherwise, the conflict is treated as the
malicious attack attempt and will not be taken into account;
verification for the new incoming transaction should continue.

This commint implements the scheme described at
https://github.com/neo-project/neo/pull/2818#issuecomment-1632972055,
thanks to @shargon for digging.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
2023-07-21 17:27:47 +03:00
Anna Shaleva
68b9ff1f17 mempool: adjust the rule of conflicting transaction ranking
Pay for all the conflicts if you'd like to went in. Close #3028.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
2023-06-01 13:33:21 +03:00
Anna Shaleva
3a71aafc43 core: distinguish notarypool/mempool metrics
Move them to the core/network packages, close #2950. The name of
mempool's unsorted transactions metrics has been changed along the
way to match the core's metrics naming convention.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
2023-04-13 18:40:19 +03:00
Anna Shaleva
6b21ad9922 *: replace interface{} with any keyword
Everywhere including examples, external interop APIs, bindings generators
code and in other valuable places. A couple of `interface{}` usages are
intentionally left in the CHANGELOG.md, documentation and tests.
2023-04-04 13:22:42 +03:00
Elizaveta Chichindaeva
28908aa3cf [#2442] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-04 19:48:27 +03:00
Roman Khimov
428ba48444 mempool: use capacity parameter to avoid map reallocations
Which is what happens in AddBlock that fills a new pool with exactly the
number of transactions we know already.
2021-12-01 21:36:25 +03:00
Roman Khimov
748a70569a mempool: use uint256 library for fee tracking
It's very effective in avoiding allocations for big.Int, we don't have a
microbenchmark for memppol, but this improves TPS metrics by ~1-2%, so it's
noticeable.
2021-12-01 21:36:25 +03:00
Evgeniy Stratonikov
e6aa9b76ee mempool: reword ErrConflict message, fix #2197
Make it more user-friendly.

Signed-off-by: Evgeniy Stratonikov <evgeniy@nspcc.ru>
2021-09-28 09:57:42 +03:00
Anna Shaleva
b8e96ac82b core: move mempool.Event to a separate package
And write a marshaller for EventType, it'll be reused by the Notification
subsystem.
2021-06-01 12:24:28 +03:00
Roman Khimov
9d2712573f *: enable godot linter and fix all its warnings
It's important for NeoGo to have clean documentation. No functional changes.
2021-05-12 23:17:03 +03:00
Anna Shaleva
bfbd096fed core: introduce mempool notifications 2021-02-02 22:01:32 +03:00
Anna Shaleva
19fa0daaa6 core, network: add Notary module 2021-02-02 22:01:20 +03:00
Anna Shaleva
09f0f03775 core: fix mempool.Add
Unlucky transaction could also have OracleResponce attribute.
2021-01-15 16:43:21 +03:00
Anna Shaleva
93a5c37696 core: fix bug with mempool.verifiedMap
Transaction is added to verifiedMap before OOM check, so we may have a
case when OOM occurs during tx1 pooling, but mp.containsKey(tx1)
returns `true` after this. Fixed.
2020-12-16 14:08:05 +03:00
Anna Shaleva
0b5cf78468 network: add notary request payload 2020-12-10 18:17:31 +03:00
Evgenii Stratonikov
cba117352c mempool: correctly handle tx with oracle response
If tx with the same oracle response ID is already in mempool,
replace it if network fee of added transaction is higher and
return error otherwise.
2020-12-01 12:29:18 +03:00
Evgenii Stratonikov
3e5b84348d network: retransmit stale transactions 2020-11-12 13:51:44 +03:00
Anna Shaleva
ec63d5c456 core: add conflicts attribute
Close #1491
2020-10-29 10:57:31 +03:00
Roman Khimov
83fc38ae3a mempool: don't create new big.Int in tryAddSendersFee() if possible
Do a little less allocations.
2020-09-10 15:35:19 +03:00
Roman Khimov
fc7ea6217d mempool: avoid reassigning utilityBalanceAndFees value
It's not needed, we're either creating a new one and assigning it 6 lines
above or we're changing already existing big.Int via a pointer, so no update
is needed.
2020-09-10 15:20:04 +03:00
Roman Khimov
fe1f1d19be mempool: store only pointer in the verifiedMap
It's only used for presence checks, there is no need for metadata here.
2020-09-09 20:46:31 +03:00
Roman Khimov
9591d64e53 mempool: don't sort items by hash
There is nothing requiring us to do so. It also is bad because it allows for
new transaction to replace some already existing one with the same fee
parameters just because it has "better" hash.

But the other thing is that for transactions with equal fees it's always
better for us to append them to the end of the list, instead of inserting them
in the middle, so this change allows to reduce slice item movements and gain
some 6-7% increase for single-node TPS.
2020-09-09 20:46:31 +03:00
Roman Khimov
5df726db68 mempool: replace timeStamp with blockStamp
Time is not really relevant for us here and we don't use this timestamp in any
way. Yet it occupies 24 bytes and we do two clock_gettime calls to get it.

Replace it with blockStamp which is going to be used in the future for
transaction retransmissions.

It allows to improve single-node TPS by another 3%.
2020-09-09 20:46:31 +03:00
Roman Khimov
0ea8c8ba67 mempool: drop a level of indirection
`item` is so small that it makes no sense bothering memory allocator with
every instance of it.
2020-09-09 20:46:31 +03:00
Evgenii Stratonikov
2661ebd295 transaction: add HighPriority attribute
HighPriority attributes specifies that transaction was
signed by a committee.
2020-08-23 09:39:46 +03:00
Roman Khimov
0e086d61ac mempool: store feeSum as big.Int
Prevent (very) potential overflow.
2020-08-20 19:06:59 +03:00
Roman Khimov
55b2cbb74d core: refactor and improve verification and pooling
Now we have VerifyTx() and PoolTx() APIs that either verify transaction in
isolation or verify it against the mempool (either the primary one or the one
given) and then add it there. There is no possibility to check against the
mempool, but not add a transaction to it, but I doubt we really need it.

It allows to remove some duplication between old PoolTx and verifyTx where
they both tried to check transaction against mempool (verifying first and then
adding it). It also saves us utility token balance check because it's done by
the mempool anyway and we no longer need to do that explicitly in verifyTx.

It makes AddBlock() and verifyBlock() transaction's checks more correct,
because previously they could miss that even though sender S has enough
balance to pay for A, B or C, he can't pay for all of them.

Caveats:
 * consensus is running concurrently to other processes, so things could
   change while verifyBlock() is iterating over transactions, this will be
   mitigated in subsequent commits

Improves TPS value for single node by at least 11%.

Fixes #667, fixes #668.
2020-08-20 18:50:18 +03:00
Roman Khimov
0d8cc437fe mempool: swap checks in Add, fail fast
Checking for duplicates is easier than checking the balance, so it should be
done first.
2020-08-20 18:50:18 +03:00
Roman Khimov
e998c102ca mempool: rename NewMemPool into New
"mempool" is a package name already.
2020-08-20 18:50:18 +03:00
Anna Shaleva
90825efa16 core: move transaction's sender to cosigners
Closes #1184

Ported changes from https://github.com/neo-project/neo/pull/1752
2020-08-04 17:33:50 +03:00
Anna Shaleva
789ee2d3c1 core: do not update mempool while verifying tx
Closes #1168
2020-07-09 20:45:59 +03:00
Anna Shaleva
4be1009def core: refactor checkBalanceAndUpdate 2020-07-09 18:27:20 +03:00
Anna Shaleva
abe3c94b95 core: use big.Int to store NEP5 balances
closes #1133
2020-07-09 13:26:39 +03:00
Anna Shaleva
73b630db9b *: switch from fixed8 to int64
Follow C# implementation, we have to marshall JSON Fixed8 fields without
taking into account decimals.
2020-06-29 21:39:27 +03:00
Anna Shaleva
9f11a55bd6 core: add policy check to mempool
After block was stored it's possible to have new FeePerByte constraint,
so we should remove all transactions which do not meet this requirement.
Also caching of FeePerByte was added in order not to re-verify
transactions each time mempool needs to be updated.
2020-06-24 07:58:09 +03:00
Evgenii Stratonikov
5354352d63 core: remove transaction priority
There is no such thing as high/low priority transactions, as there are
no free transactions anymore and they are ordered by fees contained
in transaction itself.

Closes #1063.
2020-06-18 22:44:10 +03:00
Roman Khimov
63f4f34659 mempool: drop TxWithFee type
All the fees are in transaction, this makes no sense.
2020-06-05 19:21:40 +03:00
Roman Khimov
709146f295 transaction: drop Inputs and Outputs, forget UTXO 2020-06-05 19:20:16 +03:00
Roman Khimov
169c5ae775 transaction: drop Issue TX support 2020-06-05 19:20:16 +03:00
Roman Khimov
dfc7a9bfd1 transaction: drop Claim TX type 2020-06-05 19:20:16 +03:00
Anna Shaleva
1afd630169 core: add senders' fees monitor to mempool
In order to simplify maintainance of overall current senders` fees
in the mempool we are to keep them in a separate map.
2020-05-20 23:34:25 +03:00
Anna Shaleva
7acf5b2841 core: add SystemFee and NetworkFee to transaction
closes #831
2020-05-20 23:26:48 +03:00
Roman Khimov
ec76a0bf15 mempool: disallow more than one issue tx at once
Technically they could conflict for available asset amount, but as they're
very rare (and even can be considered obsolete) we can simplify this check.
2020-03-17 11:40:01 +03:00