8f61cc1dcc
[ #5 ] policer: Use generic LRU client
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
55b403e0ee
[ #5 ] morph/client: Use generic LRU cache
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
0272218eb9
[ #2184 ] compression: Properly calculate upper bound
...
If the data is not compressible allocating `len(data)` will lead to a
slice reallocation. For a compressible data the results for small size
are flaky and we allocate a bit more. However, it feels right to use a
provided function if we need to pick any size at all.
```
name old time/op new time/op delta
Compression/size=128/zeroed_slice-8 2.23µs ±12% 2.06µs ± 6% -7.35% (p=0.009 n=10+10)
Compression/size=128/not_so_random_slice_(block_=_123)-8 19.0µs ±10% 15.8µs ±16% -17.09% (p=0.000 n=9+10)
Compression/size=128/random_slice-8 17.6µs ±15% 16.1µs ±16% ~ (p=0.075 n=10+10)
Compression/size=1024/zeroed_slice-8 3.05µs ±11% 2.84µs ±10% ~ (p=0.089 n=10+10)
Compression/size=1024/not_so_random_slice_(block_=_123)-8 18.1µs ± 6% 18.2µs ±12% ~ (p=0.971 n=10+10)
Compression/size=1024/random_slice-8 48.6µs ± 6% 45.6µs ± 5% -6.07% (p=0.006 n=10+9)
Compression/size=32768/zeroed_slice-8 26.8µs ± 3% 28.7µs ± 8% +7.23% (p=0.001 n=10+10)
Compression/size=32768/not_so_random_slice_(block_=_123)-8 44.3µs ± 8% 43.7µs ±13% ~ (p=0.762 n=8+10)
Compression/size=32768/random_slice-8 97.3µs ±32% 68.9µs ±15% -29.13% (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8 29.8ms ± 9% 30.3ms ±17% ~ (p=1.000 n=9+9)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8 33.1ms ±14% 30.3ms ±11% -8.61% (p=0.043 n=10+10)
Compression/size=33554432/random_slice-8 41.7ms ± 3% 30.1ms ± 8% -27.72% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
Compression/size=128/zeroed_slice-8 128B ± 0% 144B ± 0% +12.50% (p=0.000 n=10+10)
Compression/size=128/not_so_random_slice_(block_=_123)-8 384B ± 0% 144B ± 0% -62.50% (p=0.000 n=10+10)
Compression/size=128/random_slice-8 384B ± 0% 144B ± 0% -62.50% (p=0.000 n=10+10)
Compression/size=1024/zeroed_slice-8 1.02kB ± 0% 1.15kB ± 0% +12.50% (p=0.000 n=10+10)
Compression/size=1024/not_so_random_slice_(block_=_123)-8 1.02kB ± 0% 1.15kB ± 0% +12.50% (p=0.000 n=10+10)
Compression/size=1024/random_slice-8 2.56kB ± 0% 1.15kB ± 0% -55.00% (p=0.000 n=10+10)
Compression/size=32768/zeroed_slice-8 32.8kB ± 0% 41.0kB ± 0% +25.00% (p=0.000 n=10+10)
Compression/size=32768/not_so_random_slice_(block_=_123)-8 32.8kB ± 0% 41.0kB ± 0% +25.00% (p=0.000 n=10+10)
Compression/size=32768/random_slice-8 81.9kB ± 0% 41.0kB ± 0% -50.00% (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8 33.6MB ± 0% 33.6MB ± 0% +0.02% (p=0.000 n=9+9)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8 33.6MB ± 0% 33.6MB ± 0% +0.02% (p=0.000 n=8+10)
Compression/size=33554432/random_slice-8 75.5MB ± 0% 33.6MB ± 0% -55.55% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Compression/size=128/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal)
Compression/size=128/not_so_random_slice_(block_=_123)-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
Compression/size=128/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
Compression/size=1024/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal)
Compression/size=1024/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal)
Compression/size=1024/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
Compression/size=32768/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal)
Compression/size=32768/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal)
Compression/size=32768/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal)
Compression/size=33554432/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
741482c26f
[ #2164 ] neofs-node: Allow to set reconnect interval
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
6f5edac730
[ #2164 ] network/cache: Do not reconnect to failed clients immediately
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f3caf6acfe
[ #2164 ] network/cache: Separate mutex for addr
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0ace28e43d
[ #2175 ] blobovniczatree: Close all non-active blobovniczas
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c1cf418956
[ #2175 ] blobovniczatree: Make function parameters more descriptive
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
b4e90cdf51
[ #2165 ] pilorama: Optimize TreeApply when used for synchronization
...
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f9fcd85363
[ #2165 ] services/tree: Remember starting height for the synchronization
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
06137dbf8e
[ #2165 ] services/tree: Do not export synchronizeAllTrees
...
It is used only privately.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c299b98afe
[ #2165 ] services/tree: Parallelize synchronization
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
de9957e076
[ #2165 ] services/tree: Always synchronize all containers
...
In case of split-brain we must synchronize everything.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
6a4e5e6f0a
[ #2144 ] node: Try node's private key if dynamic token fetching failed
...
`GETRANGEHASH` request spawns `GETRANGE` requests if an object could not be
found locally. If the original request contains session, it can be static
and, therefore, fetching session key can not be performed successfully.
As the best effort a node could request object's range with its own key.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
86a4fba571
[ #2144 ] node: Clarify KeyStorage.GetKey method
...
Actualize the doc, fix API status error return.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21717262ec
[ #2016 ] shard: Check meta first on Get
...
`meta` should prevent returning removed objects (`GCMark` and `TS` relations
are `meta` abstractions).
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
74ec71446f
[ #2167 ] shard: Do not use write-cache by default in Head
...
Both `meta` and `write-cache` are expected to have a fast underlying disk,
so it does not seem like an optimisation. Moreover, `write-cache`'s `Head`
is a `Get` with payload cutting, it _must_ use more memory for no reason
(`meta` was created for such requests). Also, `write-cache` does not allow
performing any "meta" relations checks (such as locking, tombstoning).
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
1608fd1c07
[ #2167 ] write-cache: Add "write-cache" to its logs
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
eea2892109
[ #1956 ] node: Lock shard's mode on its methods switch
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
04b5ec759b
[ #2139 ] object/put: Use sync.Pool for temporary payloads
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
9e0decd12d
[ #2162 ] services/tree: Close connection after the syncronization
...
There was a goroutine leak here.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e1c3bdbfa6
[ #1621 ] pilorama: Remove Timestamp field from nodeInfo
...
It is already present in `Meta`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
1044adbe94
[ #1621 ] pilorama: Improve memory allocation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
2539d466a6
[ #1621 ] pilorama: Seek after cursor invalidation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e9ba8931f8
[ #1621 ] pilorama: Simplify bucket creation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
fe7ddfdc6a
[ #1621 ] pilorama: Compare memory forests properly
...
Node children are not sorted and could occur in any order.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
306609030a
[ #2159 ] node: Add tree replication timeout configuration
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
edb1428248
[ #2022 ] Add metric readonly to get shards mode
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
e5c304536b
[ #2161 ] pilorama: Do not apply already existing operations
...
Speeds up synchronization a bit.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
3bb5a320d7
[ #2154 ] services/tree: Do not log an error when synchronizing container of 1 node
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
387d1e2977
[ #2127 ] services/tree: Randomize node order for synchronization
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21c58c92a9
[ #2145 ] meta: Do allow force inhuming a locked object
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
b207dc424f
[ #2158 ] policer: Reduce default cache size
...
We use cache to avoid policing the same object multiple times in a short
time span (< 30 seconds). If we have 200_000 objects in a blobstor, it is a bit useless
-- if it takes 1 second to process an object and we have `replicator.pool_size: 20`
in config, the next iteration will happen in 10_000 second which is much
larger than 30 second. However we still consume a lot of memory, so it
makes sense to use saner default.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0244f2e5ce
[ #2156 ] core/netmap: Change node addresses iteration order
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
b413094704
[ #2095 ] node: Fix collecting child objects
...
Stop child objects collection if the last returned object (the most "left"
object in the collected chain) starts exactly from the `GETRANGE`'s `from`
value.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
350eecfa13
[ #2095 ] node: Do not allow GETRANGE requests with zero length
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
0b78af467e
[ #2140 ] engine: Fix error handling in TreeMove
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a
Move to frostfs-node
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
42554a9298
[ #2068 ] writecache: Remove deleted objects from the writecache
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
4a49ea0855
[ #2068 ] writecache: Allow to open FSTree in read-only mode
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
857d2dc3f5
[ #2068 ] writecache: Optimize initial flush existence checking
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
63f604e948
[ #2068 ] blobstor: Allow to provide storage ID in Exists
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
6ad2b5d5b8
[ #2068 ] blobovnicza: Add Exists method
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
ba74244d1b
[ #1486 ] ir: Use endless notary deposit for side chain
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
b6a40241f8
[ #1486 ] morph: Add method for endless notary deposit
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
Pavel Karpy
d54022eacc
[ #2047 ] node: Do not send chunk twice on request forwarding
...
That could happen if a node forwards request to a node that closed the
connection during the original object stream.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-02 11:27:48 +03:00
Evgenii Stratonikov
bd25db5d4a
[ #1984 ] metrics: Use separate metrics for success/failed requests
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-01 14:18:10 +03:00
Evgenii Stratonikov
e21c472dc7
[ #1984 ] services/object: Increase put_req_count after the request is processed
...
As it is specified in metrics description.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-01 14:18:10 +03:00
Pavel Karpy
50d28b72c3
[ #2075 ] node, ir: Log notary state with info
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
b8c30b88f6
[ #2075 ] morph: Do not do notary deposit with zero balance
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
c01d4ecb50
[ #2080 ] morph: Close morph clients
...
Could be related to "websocket users limit reached" on the `neo-go` server
side when an SN/IR is rebooting repeatedly.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
3d0768a1d3
[ #2061 ] node: Unify meta.Get benchmarks
...
Make them get exactly one (different) object per a bench iteration.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 18:29:14 +03:00
Pavel Karpy
bc905f169d
[ #2061 ] meta: Add parallel bench for Get
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 18:29:14 +03:00
Pavel Karpy
960a2d0629
[ #2081 ] ir: Set default key in IR's SDK clients
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Pavel Karpy
51963abce7
[ #1972 ] node: Fix errors comments in the Put service
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
7335a52f29
[ #1732 ] pilorama: Improve logical error handling
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
ae7b473768
[ #2064 ] blobovniczatree: Remove index too big log
...
There is no need to log about a situation which is expected.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:53:18 +03:00
Evgenii Stratonikov
c1ea6fd854
[ #2091 ] morph/client: Use notary.FakeContractAccount wrapper
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
b93be8869b
[ #2091 ] morph/client: Simplify code
...
1. Replace `mn` function with a `sigCount`.
2. Use `notary.FakeMultisigAccount` for account creation.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
6af52c46d8
[ #2091 ] morph/client: Do not use deprecated methods
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
816c74d185
[ #2075 ] morph/client: Ignore error if a transaction already exists
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
71853348b9
[ #2063 ] morph/client: Support new hash format in morph nns client
...
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-11-19 11:01:04 +03:00
Pavel Karpy
ed4351aab0
[ #2074 ] write-cache: Do not flush same object twice
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
dd225906a0
[ #2074 ] write-cache: Remove unused variables
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
9f7ac6bf9f
[ #2069 ] innerring: Do not panic in Head
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
660c38d07e
[ #2062 ] services/policer: Use a proper key for object cache
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b1025bdb42
[ #2057 ] meta: Fail write operations in R/O mode
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fdeea1dfac
[ #2057 ] meta: Fix concurrent mode changes
...
Includes:
1. mode change read lock operation in every exported method that r/w the
underlying database;
2. returning `ErrDegradedMode` logical error if any exported method is
called in degraded (without a metabase) mode.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
3d6defd3e8
[ #2057 ] meta: Do not lock the whole meta on GET
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fa231b8c56
[ #2057 ] blobstor: Block operations on a mode change
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
1779664644
[ #2058 ] services/policer: Fix panic in shardPolicyWorker
...
```
2022/11/15 08:40:56 worker exits from a panic: runtime error: index out of range [0] with length 0
2022/11/15 08:40:56 worker exits from panic: goroutine 1188 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0x1042b60, 0xc0015ae018})
runtime/panic.go:1038 +0x215
github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x366
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x68
```
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b673d9e472
[ #2053 ] engine: Do not switch mode because of logical errors
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
9a20498f34
[ #1940 ] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-19 11:01:04 +03:00
Pavel Karpy
634792077e
[ #1502 ] node: Store lock object on every container node
...
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fe09cd9c70
[ #1502 ] core: Add AddressWithType
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
3b61cb4f49
[ #1502 ] engine: Check all shards for LOCK'ing before inhuming
...
It allows keeping all the locked objects safe after metabase
resynchronization. Currently, all `LOCK` objects are broadcast to all nodes
in a container, it guarantees `LOCK` object presence in a regular situation.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
34e8d2ba56
[ #1502 ] shard: Add IsLocked method
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
9a039ba582
[ #1502 ] meta: Add IsLocked method
...
It gets an object and returns its locking status.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
659011c143
[ #2048 ] network/cache: Optimize ClientCache
...
1. Remove a layer of indirection for mutex, `ClientCache` is already
used by pointer.
2. Fix duplication of a `AllowExternal` field.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
d5a14041e0
[ #2040 ] node: Do not attach tokens in the assembly process
...
A container node is expected to have full "get" access to assemble the
object.
A non-container node is expected to forward any request to a container node.
Any token is expected to be issued for an original request sender not for a
node so any new request is invalid by design with that token.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fd61bdadcb
[ #2040 ] node: Attach original meta to the spawned requests
...
Do not lose meta information of the original requests: cache session and
bearer tokens of the original request b/w a new generated ones. Middle
request wrappers should not contain any meta information, since it is
useless (e.g. ACL service checks only the original tokens).
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d65604ad30
[ #1985 ] blobstor: Allow to report multiple errors to caller
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
f2d7e65e39
[ #2035 ] engine: Allow moving to degraded from background workers
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
481b48b942
[ #2028 ] node: Check session token's NBF and IAT
...
ACL service did not check "Not Valid Before" and "Issued At" claims.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aadd2ad050
[ #2028 ] node: Do not wrap malformed request errors
...
After presenting request statuses on the API level, all the errors are
unwrapped before sending to the caller side. It led to a losing invalid
request's context.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
2522d924b9
[ #2037 ] services/object: Fix concurrent map writes in traverser
...
```
fatal error: concurrent map writes
goroutine 4337 [running]:
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*traversal).submitProcessed(...)
github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:78
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*distributedTarget).iteratePlacement.func1()
github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:198 +0x265
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
```
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
b0e94b6a6b
[ #1906 ] writecache: Do not require read-only mode in Flush
...
It was needed before we started to flush during transition to
`degraded` mode. Now it is confusing.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
aa478f1def
[ #2024 ] services/object: Unify status errors
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
3875fef542
[ #2024 ] services/object: Cover corner cases for children OutOfRange
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b9fb0d6050
[ #1972 ] node: Fix object format unit tests
...
Includes:
1. Unused func removal;
2. Err check of the `Sign` method.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aab398f4f5
[ #1972 ] node: Do not save objects if node not in a container
...
Do not use node's local storage if it is clear that an object will be
removed anyway as a redundant. It requires moving the changing local storage
logic from the validation step to the local target implementation.
It allows performing any relations checks (e.g. object locking) only if a
node is considered as a valid container member and is expected to store
(stored previously) all the helper objects (e.g. `LOCK`, `TOMBSTONE`, etc).
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
2849e465f9
[ #1699 ] meta: Do not return SplitInfoError on Delete
...
It is not an error: removing virtual object is expected and should be just
skipped. Getting a virtual object with `raw` flag is considered as an
impossible action, all the virtual objects removals will be handled via
their children's removals implicitly.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a3e7365cbd
[ #1732 ] pilorama: Fill parent mark correctly
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
134f2ba02e
[ #1732 ] pilorama: Fix backwards log insertion
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
2ef38cfbc4
[ #1996 ] engine: Ignore pilorama.ErrTreeNotFound for write operations
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a455ec18c3
[ #2007 ] services/object: Allocate memory on-demand in GET_RANGE
...
For big objects we want to get OutOfRange error before all the memory is
allocated.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
ff5526038d
[ #2007 ] services/object: Fix comment
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d8d3588e1b
[ #1996 ] engine: Always select proper shard for a tree
...
Currently there is a possibility for modifying operations to fail
because of I/O errors and a new tree to be created on another shard.
This commit adds existence check for modifying operations.
Read operations remain as they are, not to slow things.
`TreeDrop` is an exception, because this is a tree removal and trying
multiple shards is not an unwanted behaviour.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-03 15:29:23 +03:00
Evgenii Stratonikov
777fd32d4f
[ #1818 ] writecache: Increase error counter on background errors
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
bffb0f894c
[ #1818 ] writecache: Update storage ID during flush
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
