4578d00619
[ #321 ] shard/test: Execute tests in parallel
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
d35e4c389f
[ #321 ] shard/test: Parallelize TestWriteCacheObjectLoss
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
969bfb603f
[ #321 ] shard/test: Parallelize TestShard_List
...
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After: 0.109s
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
a181c9e434
[ #332 ] gc: Add additional logging
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 17:34:40 +03:00
973af12854
[ #327 ] tests: replace os.MkdirTemp with t.TempDir
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-05 11:24:51 +00:00
a1b4ba9980
[ #209 ] compression: Do not store uncompressible data
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
eca5c210dd
[ #299 ] evacuate: Add context cancel checks
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 15:55:30 +03:00
a9c4ba62c3
[ #248 ] metabase: Remove storage group bucket
...
Backwards compatible change, so no version increase.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
58f1ba4b51
[ #288 ] pilorama: Add missing operation in log
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
daa26f6e9b
[ #288 ] pilorama/test: Check operation order for TreeGetByPath()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
291f9e809a
[ #288 ] pilorama: Remove getMeta() wrapper
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
0045f1bcd4
[ #288 ] pilorama: Use more descriptive names for memory tree
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
f856ad7480
[ #288 ] pilorama: Remove childMap from memory forest
...
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
ada081dfd5
[ #19 ] node: Make policier read shards concurrently
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
* Introduce ListWithMultiCursor that simultaneously reads objects
from different shards
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-27 11:11:47 +03:00
1f4061c0e2
[ #285 ] blobonicza: Optimize upperPowerOfTwo()
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
revive empty-block: this block is empty, you can remove it
```
Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8 0.4924n ± 1% 0.2472n ± 2% -49.80% (p=0.000 n=10)
UpperBound/size=1023-8 0.4936n ± 3% 0.2442n ± 1% -50.52% (p=0.000 n=10)
UpperBound/size=66560-8 0.8201n ± 2% 0.2436n ± 1% -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8 6.6900n ± 5% 0.2432n ± 0% -96.36% (p=0.000 n=10)
geomean 1.075n 0.2446n -77.24%
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:11 +03:00
f07e2d4812
[ #285 ] lint: Fix revive/unused-parameter
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
Pavel Karpy
09938a9841
Revert "[ #262 ] meta: Do not return old expired objects"
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
This reverts commit 3d23b087
2023-04-20 19:22:55 +03:00
Pavel Karpy
e9461686b8
[ #274 ] wc: Resolve possible deadlock
...
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:54 +03:00
Pavel Karpy
6b6f33ed71
[ #274 ] wc: Make wait groups work more explicit
...
Do not run routine that calls `wg.Done()` inside, it is hard to read.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:54 +03:00
6c90bb87f1
[ #118 ] node: add ctx for unit tests for blobstor
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 18:09:33 +03:00
Pavel Karpy
3d23b08773
[ #262 ] meta: Do not return old expired objects
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-19 13:37:38 +00:00
13c8afcb02
[ #118 ] node: add unit concurrent tests for blobstor
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 10:22:50 +00:00
Pavel Karpy
20cd080323
[ #255 ] write-cache: Fix init race condition
...
ci/woodpecker/push/pre-commit Pipeline was successful
Do not use WC's internals in the initialization routines without mode
protection. WC should be able to change its mode even if the initialization
is not finished yet.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-19 09:29:07 +00:00
Pavel Karpy
262c9c2b93
[ #256 ] blobovniczaTree: Make Exists test stable
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Corrupt and request _the same_ file.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 16:46:24 +03:00
8466894fdf
[ #250 ] control: remove DumpShard and RestoreShard RPC
...
ci/woodpecker/push/pre-commit Pipeline was successful
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 12:28:49 +00:00
6121b541b5
[ #242 ] treesvc: Add tracing spans
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
d62c6e4ce6
[ #242 ] node: Add tracing spans
...
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
41eb3129ae
[ #139 ] Refactor blobovnicza exist test to not use chmod
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-14 12:16:14 +03:00
0e31c12e63
[ #240 ] logs: Move log messages to constants
...
Drop duplicate entities.
Format entities.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
5d2affa5cd
testutil: Fix linter warning
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 11:00:02 +03:00
0920d848d0
[ #135 ] get-object: Add tracing spans
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
c4865783fc
[ #236 ] blobstor/test: Prefill storage in parallel in read benchmark
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:11 +03:00
6ad5c38225
[ #236 ] testutil: Use random object id in RandObjGenerator
...
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8 227425 4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
c85a0bc866
[ #236 ] blobstor/test: Reduce test descriptions
...
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
dbc3811ff4
[ #191 ] engine: Allow to remove redundant object copies
...
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
cb172e73a6
[ #228 ] node: Use uber atomic package instead standard
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 15:37:27 +00:00
f32f61df87
[ #203 ] pilorama: Refactor tests
...
Do not pass 0 as channel capacity.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
8908798f59
[ #203 ] node: Resolve unused vars
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
bab11492ad
[ #203 ] node: Resolve never used errors
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
9e2df4b7c7
[ #203 ] node: Fix double imports
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
ab891517de
[ #116 ] node: Fix bug with extra generated files in TestReload
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
* Create testNewEngine in engineWithShards without default opts
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 18:49:34 +03:00
6f7b6a8813
[ #116 ] node: Improve shard/engine construction in tests
...
ci/woodpecker/push/pre-commit Pipeline was successful
* Introduce testEngineWrapper that can be constructed with different options
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 14:36:40 +00:00
23575e1ac0
[ #210 ] policier: Resolve contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
9098d0eec0
[ #211 ] engine: Unify shard mode checks for tree operations
...
ci/woodpecker/push/pre-commit Pipeline was successful
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
760af6b912
[ #211 ] fstree: Consider ENOSPC a logical error
...
We already do this for file writing, however directory creation can also
fail.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
1f1aed87be
[ #188 ] metabase: Refactor object inhume
...
ci/woodpecker/push/pre-commit Pipeline was successful
Resolve funlen linter for db.Inhume method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
5a66db80c5
[ #188 ] engine: Refactor shard evacuation
...
Resolve funlen and gocognit linter for StorageEngine.Evacuate method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
456bc097f7
[ #188 ] engine: Refactor get range from engine
...
Resolve funlen linter for StorageEngine.getRange method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
3010ca2649
[ #188 ] engine: Refactor get object from engine
...
Resolve funlen linter for StorageEngine.get method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
0739c36a3b
[ #188 ] metabase: Refactor object put to metabase
...
Resolve funlen linter for db.put method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
8273a3dfb2
[ #188 ] blobstor: Refactor blobstor test
...
Resolve funlen linter for TestIterate function
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
594b5821ed
[ #188 ] blobstor: Refactor put data to blobovniczas
...
Resolve funlen linter for Blobovniczas.Put method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
8e5a0dcf27
[ #204 ] gc: Fix GC handlers start
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 06:48:27 +00:00
a7c79c773a
[ #168 ] node: Refactor node config
...
Resolve containedctx linter for cfg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
7be5a0fd79
[ #189 ] node: Fix bug with sync/atomic package
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-30 14:50:43 +00:00
ab32067152
[ #183 ] gc: Fix drop expired locked complex objects
...
ci/woodpecker/push/pre-commit Pipeline was successful
Do not delete bucket keys during iteration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
9f0bce5c15
[ #183 ] gc: Fix drop expired locked simple objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
341fe1688f
[ #139 ] test: Add test storage implementation
...
ci/woodpecker/push/pre-commit Pipeline was successful
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-29 14:28:49 +00:00
Pavel Karpy
97e201993b
[ #184 ] *: Unify done contexts handling
...
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
221203beeb
[ #180 ] node: Refactor panics in unit test
...
ci/woodpecker/push/pre-commit Pipeline was successful
* Replace panics in unit tests by require.NoError and t.Fatalf
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-29 12:39:07 +03:00
aarifullin
34329d67ff
[ #86 ] node: Fix unit test and linter errors
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 12:42:58 +03:00
9808dec591
[ #86 ] node: Move testing utils to one package
...
ci/woodpecker/push/pre-commit Pipeline failed
Move testing utils from tests in local_object_storage package to
unified testutil package
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 08:19:15 +00:00
342e571d89
[ #159 ] Add handle __SYSTEM__ sys attributes
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-22 17:35:20 +03:00
Pavel Karpy
da8da1c63a
[ #98 ] fstree: Do not fail iteration over just removed files
...
ci/woodpecker/push/pre-commit Pipeline was successful
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133)
2023-03-22 07:26:27 +00:00
8014fdb21a
[ #156 ] metabase: Make freePotentialLocks() pass linter checks
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
fb13902db9
[ #156 ] shard: Make refillMetabase() pass linter checks
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
47e8c5bf23
[ #156 ] pilorama: Remove CIDDescriptor from TreeApply()
...
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
only used in controlled environments.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
5059dcc19d
[ #145 ] shard-gc: Delete expired objects after locks
...
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
6c4a1699ef
[ #145 ] shard-gc: Expired locked unit test
...
Added unit test that verifies that GC deletes expired
locked objects in one epoch.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
481a1ca6f3
[ #148 ] linter: Add gocognit linter
...
Code with high cognitive complexity is hard intuitively to understand
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
97c36ed3ec
[ #148 ] linter: Add funlen linter
...
Long functions are hard to understand and source of errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
2dc86058c3
[ #148 ] memstore: Drop space line
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
Pavel Karpy
f006f3b342
[ #67 ] node: Make engine's IsLocked public
...
It will allow reusing that method in expiration checks.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:20:45 +03:00
724debfdcd
[ #81 ] node: Add basic read/write benchmarks for substorages
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-15 16:37:04 +00:00
3e6fd4c611
[ #82 ] pilorama: Allow to store last sync height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
861e9ab59a
[ #83 ] pre-commit: Add initial configuration
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
Pavel Karpy
f1f3c80dbf
[ #32 ] node: Init write-cache asynchronously
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
Pavel Karpy
381e363a8b
[ #32 ] node: Always close general components after testing
...
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
e9f3c24229
[ #65 ] Use strings.Cut instead of strings.Split* where possible
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:39:14 +03:00
6925fb4c59
[ TrueCloudLab/hrw#2 ] node: Use typed HRW methods
...
Update HRW lib and use typed HRW methods to sort shards and nodes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
c3a7039801
[ TrueCloudLab/hrw#2 ] node: Optimize shard hash
...
Compute shard hash only once
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
cb5468abb8
[ #66 ] node: Replace interface{} with any
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-21 16:47:07 +03:00
Pavel Karpy
337049b2ce
[ #56 ] node: Allow reading expired locked object
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-21 09:56:57 +03:00
Pavel Karpy
3beef10f89
[ #61 ] node: Do not fetch missing objects
...
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 14:47:38 +03:00
d1d123d180
[ #2234 ] writecache: Fix possible panic in initFlushMarks
...
In case we have many small objects in the write-cache, `indices` should
not be reused between iterations.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
315141dc2c
[ #2252 ] fstree: Allow concurrent writes
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
07ec51ea60
[ #2244 ] node: Add object address to WC's operations
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
dbbbef9ddb
[ #2244 ] node: Update expired storage ID by WC
...
Previously, node could get an "infinite" small object: it could be expired
and thus could not be flushed (update its storage ID) to metabase => could
not be marked as flushed => node never removes such object and repeat all
the cycle one more time. If object exists and is not marked with GC (meta
returns `ErrObjectIsExpired`, not `ObjectNotFound` and not
`ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt
transaction.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
5cb2c5ae62
[ #2238 ] engine: Add test for component initialization failures
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
427fe276f2
[ #2238 ] shard: Try closing all components
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
c53903ccd0
[ #2238 ] engine: Make Open and Init similar
...
1. Both could initialize shards in parallel.
2. Both should close shards after an error.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
e0309e398c
[ #2239 ] writecache: Fix possible deadlock
...
LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction.
`View` transaction itself takes `mmapLock` [1], which is lifted after tx
finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`)
When we evict items from LRU cache mutex order is different:
first we take LRU mutex and then execute `Batch` which _does_ take
`mmapLock` in case we need to remap. Thus the deadlock.
[1] 8f4a7e1f92/db.go (L708)
2023-02-20 13:53:27 +03:00
58367e4df6
[ #2232 ] pilorama: Merge in-queue batches
...
To achieve high performance we must choose proper values for both
batch size and delay. For user operations we want to set low delay.
However it would prevent tree synchronization operations to form big
enough batches. For these operations, batching gives the most benefit
not only in terms of on-CPU execution cost, but also by speeding up
transaction persist (`fsync`).
In this commit we try merging batches that are already
_triggered_, but not yet _started to execute_. This way we can still
query batches for execution after the provided delay while also allowing
multiple formed batches to execute faster.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
40822adb51
[ #2213 ] node: Do not return object expired object
...
"Object is expired" means that object is presented in `meta` but it is not
`ObjectNotFound` error. Previous implementation made `shard` search for an
object without `meta` which was an error.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
362f24953a
[ #47 ] shard: Switch container size metric from physical to logical capacity
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-17 12:03:42 +03:00
204cd3a11c
[ #31 ] fstree: Optimize treePath
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
dee4498c1e
[ #31 ] fstree: Do not check for a file existence twice
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
abbecf49d6
[ #31 ] fstree: Speedup string-to-address conversion
...
```
name old time/op new time/op delta
_addressFromString-8 1.25µs ±30% 1.02µs ± 6% -18.49% (p=0.000 n=9+9)
name old alloc/op new alloc/op delta
_addressFromString-8 352B ± 0% 256B ± 0% -27.27% (p=0.000 n=9+10)
name old allocs/op new allocs/op delta
_addressFromString-8 6.00 ± 0% 4.00 ± 0% -33.33% (p=0.000
n=10+10)
```
Also, assure compiler that `s` doesn't escape:
Before this commit:
```
./fstree.go:74:24: leaking param: s
./fstree.go:90:6: moved to heap: addr
```
After this commit:
```
./fstree.go:74:24: s does not escape
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
ab21d90cfb
[ #1794 ] shard: Add increasing case for the payload size metric
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-09 13:30:23 +03:00
