0920d848d0
[ #135 ] get-object: Add tracing spans
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
c4865783fc
[ #236 ] blobstor/test: Prefill storage in parallel in read benchmark
...
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:11 +03:00
6ad5c38225
[ #236 ] testutil: Use random object id in RandObjGenerator
...
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8 227425 4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
perf_test.go:165: writing entry: file exists
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
c85a0bc866
[ #236 ] blobstor/test: Reduce test descriptions
...
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
dbc3811ff4
[ #191 ] engine: Allow to remove redundant object copies
...
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
cb172e73a6
[ #228 ] node: Use uber atomic package instead standard
...
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 15:37:27 +00:00
f32f61df87
[ #203 ] pilorama: Refactor tests
...
Do not pass 0 as channel capacity.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
8908798f59
[ #203 ] node: Resolve unused vars
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
bab11492ad
[ #203 ] node: Resolve never used errors
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
9e2df4b7c7
[ #203 ] node: Fix double imports
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
ab891517de
[ #116 ] node: Fix bug with extra generated files in TestReload
...
* Create testNewEngine in engineWithShards without default opts
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 18:49:34 +03:00
6f7b6a8813
[ #116 ] node: Improve shard/engine construction in tests
...
* Introduce testEngineWrapper that can be constructed with different options
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 14:36:40 +00:00
23575e1ac0
[ #210 ] policier: Resolve contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
9098d0eec0
[ #211 ] engine: Unify shard mode checks for tree operations
...
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
760af6b912
[ #211 ] fstree: Consider ENOSPC a logical error
...
We already do this for file writing, however directory creation can also
fail.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
1f1aed87be
[ #188 ] metabase: Refactor object inhume
...
Resolve funlen linter for db.Inhume method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
5a66db80c5
[ #188 ] engine: Refactor shard evacuation
...
Resolve funlen and gocognit linter for StorageEngine.Evacuate method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
456bc097f7
[ #188 ] engine: Refactor get range from engine
...
Resolve funlen linter for StorageEngine.getRange method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
3010ca2649
[ #188 ] engine: Refactor get object from engine
...
Resolve funlen linter for StorageEngine.get method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
0739c36a3b
[ #188 ] metabase: Refactor object put to metabase
...
Resolve funlen linter for db.put method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
8273a3dfb2
[ #188 ] blobstor: Refactor blobstor test
...
Resolve funlen linter for TestIterate function
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
594b5821ed
[ #188 ] blobstor: Refactor put data to blobovniczas
...
Resolve funlen linter for Blobovniczas.Put method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
8e5a0dcf27
[ #204 ] gc: Fix GC handlers start
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 06:48:27 +00:00
a7c79c773a
[ #168 ] node: Refactor node config
...
Resolve containedctx linter for cfg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
7be5a0fd79
[ #189 ] node: Fix bug with sync/atomic package
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-30 14:50:43 +00:00
ab32067152
[ #183 ] gc: Fix drop expired locked complex objects
...
Do not delete bucket keys during iteration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
9f0bce5c15
[ #183 ] gc: Fix drop expired locked simple objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
341fe1688f
[ #139 ] test: Add test storage implementation
...
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-29 14:28:49 +00:00
Pavel Karpy
97e201993b
[ #184 ] *: Unify done contexts handling
...
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
221203beeb
[ #180 ] node: Refactor panics in unit test
...
* Replace panics in unit tests by require.NoError and t.Fatalf
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-29 12:39:07 +03:00
aarifullin
34329d67ff
[ #86 ] node: Fix unit test and linter errors
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 12:42:58 +03:00
9808dec591
[ #86 ] node: Move testing utils to one package
...
Move testing utils from tests in local_object_storage package to
unified testutil package
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 08:19:15 +00:00
342e571d89
[ #159 ] Add handle __SYSTEM__ sys attributes
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-22 17:35:20 +03:00
Pavel Karpy
da8da1c63a
[ #98 ] fstree: Do not fail iteration over just removed files
...
ci/woodpecker/push/pre-commit Pipeline was successful
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133)
2023-03-22 07:26:27 +00:00
8014fdb21a
[ #156 ] metabase: Make freePotentialLocks() pass linter checks
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
fb13902db9
[ #156 ] shard: Make refillMetabase() pass linter checks
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
47e8c5bf23
[ #156 ] pilorama: Remove CIDDescriptor from TreeApply()
...
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
only used in controlled environments.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
5059dcc19d
[ #145 ] shard-gc: Delete expired objects after locks
...
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
6c4a1699ef
[ #145 ] shard-gc: Expired locked unit test
...
Added unit test that verifies that GC deletes expired
locked objects in one epoch.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
481a1ca6f3
[ #148 ] linter: Add gocognit linter
...
Code with high cognitive complexity is hard intuitively to understand
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
97c36ed3ec
[ #148 ] linter: Add funlen linter
...
Long functions are hard to understand and source of errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
2dc86058c3
[ #148 ] memstore: Drop space line
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
Pavel Karpy
f006f3b342
[ #67 ] node: Make engine's IsLocked public
...
It will allow reusing that method in expiration checks.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:20:45 +03:00
724debfdcd
[ #81 ] node: Add basic read/write benchmarks for substorages
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-15 16:37:04 +00:00
3e6fd4c611
[ #82 ] pilorama: Allow to store last sync height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
861e9ab59a
[ #83 ] pre-commit: Add initial configuration
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
Pavel Karpy
f1f3c80dbf
[ #32 ] node: Init write-cache asynchronously
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
Pavel Karpy
381e363a8b
[ #32 ] node: Always close general components after testing
...
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
e9f3c24229
[ #65 ] Use strings.Cut instead of strings.Split* where possible
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:39:14 +03:00
