TrueCloudLab
/
frostfs-node
17
1
Fork 21
Code Issues 86 Pull Requests 6 Actions Projects Releases Activity
3,808 Commits
7 Branches
24 Tags
49 MiB
Commit Graph

803 Commits (a5f118a987c6df52a3dbda19c0a0f7fe151516eb)

Author SHA1 Message Date
Dmitrii Stepanov a181c9e434 [#332] gc: Add additional logging
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-10 17:34:40 +03:00
Alejandro Lopez 973af12854 [#327] tests: replace os.MkdirTemp with t.TempDir
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-05 11:24:51 +00:00
Evgenii Stratonikov a1b4ba9980 [#209] compression: Do not store uncompressible data 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-03 16:28:29 +03:00
Dmitrii Stepanov eca5c210dd [#299] evacuate: Add context cancel checks
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-02 15:55:30 +03:00
Evgenii Stratonikov a9c4ba62c3 [#248] metabase: Remove storage group bucket 
Backwards compatible change, so no version increase.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 17:00:43 +03:00
Evgenii Stratonikov 58f1ba4b51 [#288] pilorama: Add missing operation in log
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov daa26f6e9b [#288] pilorama/test: Check operation order for TreeGetByPath() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov 291f9e809a [#288] pilorama: Remove getMeta() wrapper 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov 0045f1bcd4 [#288] pilorama: Use more descriptive names for memory tree 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov f856ad7480 [#288] pilorama: Remove `childMap` from memory forest 
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Airat Arifullin ada081dfd5 [#19] node: Make policier read shards concurrently
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Introduce ListWithMultiCursor that simultaneously reads objects
  from different shards

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-27 11:11:47 +03:00
Evgenii Stratonikov 1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo()
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:11 +03:00
Evgenii Stratonikov f07e2d4812 [#285] lint: Fix revive/unused-parameter 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:04 +03:00
Pavel Karpy 09938a9841 Revert "[#262] meta: Do not return old expired objects"
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
This reverts commit 3d23b087

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:55 +03:00
Pavel Karpy e9461686b8 [#274] wc: Resolve possible deadlock 
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Pavel Karpy 6b6f33ed71 [#274] wc: Make wait groups work more explicit 
Do not run routine that calls `wg.Done()` inside, it is hard to read.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Airat Arifullin 6c90bb87f1 [#118] node: add ctx for unit tests for blobstor
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-19 18:09:33 +03:00
Pavel Karpy 3d23b08773 [#262] meta: Do not return old expired objects
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-19 13:37:38 +00:00
Airat Arifullin 13c8afcb02 [#118] node: add unit concurrent tests for blobstor
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-19 10:22:50 +00:00
Pavel Karpy 20cd080323 [#255] write-cache: Fix init race condition
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Do not use WC's internals in the initialization routines without mode
protection. WC should be able to change its mode even if the initialization
is not finished yet.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-19 09:29:07 +00:00
Pavel Karpy 262c9c2b93 [#256] blobovniczaTree: Make `Exists` test stable
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Corrupt and request _the same_ file.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-14 16:46:24 +03:00
Evgenii Stratonikov 8466894fdf [#250] control: remove `DumpShard` and `RestoreShard` RPC
ci/woodpecker/push/pre-commit Pipeline was successful Details 
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 12:28:49 +00:00
Dmitrii Stepanov 6121b541b5 [#242] treesvc: Add tracing spans
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Dmitrii Stepanov d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Alejandro Lopez 41eb3129ae [#139] Refactor blobovnicza exist test to not use chmod
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-04-14 12:16:14 +03:00
Evgenii Stratonikov 0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Evgenii Stratonikov 5d2affa5cd testutil: Fix linter warning 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-12 11:00:02 +03:00
Dmitrii Stepanov 0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Evgenii Stratonikov c4865783fc [#236] blobstor/test: Prefill storage in parallel in read benchmark
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:11 +03:00
Evgenii Stratonikov 6ad5c38225 [#236] testutil: Use random object id in `RandObjGenerator` 
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8            227425              4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8     --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8            --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:04 +03:00
Evgenii Stratonikov c85a0bc866 [#236] blobstor/test: Reduce test descriptions 
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:04 +03:00
Evgenii Stratonikov dbc3811ff4 [#191] engine: Allow to remove redundant object copies 
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-07 17:25:50 +00:00
Airat Arifullin cb172e73a6 [#228] node: Use uber atomic package instead standard
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 15:37:27 +00:00
Dmitrii Stepanov f32f61df87 [#203] pilorama: Refactor tests 
Do not pass 0 as channel capacity.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov 8908798f59 [#203] node: Resolve unused vars 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov bab11492ad [#203] node: Resolve never used errors 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov 9e2df4b7c7 [#203] node: Fix double imports 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Airat Arifullin ab891517de [#116] node: Fix bug with extra generated files in TestReload
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Create testNewEngine in engineWithShards without default opts

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-05 18:49:34 +03:00
Airat Arifullin 6f7b6a8813 [#116] node: Improve shard/engine construction in tests
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Introduce testEngineWrapper that can be constructed with different options

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-05 14:36:40 +00:00
Dmitrii Stepanov 23575e1ac0 [#210] policier: Resolve contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Evgenii Stratonikov 9098d0eec0 [#211] engine: Unify shard mode checks for tree operations
ci/woodpecker/push/pre-commit Pipeline was successful Details 
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-05 11:10:39 +00:00
Evgenii Stratonikov 760af6b912 [#211] fstree: Consider ENOSPC a logical error 
We already do this for file writing, however directory creation can also
fail.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-05 11:10:39 +00:00
Dmitrii Stepanov 1f1aed87be [#188] metabase: Refactor object inhume
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for db.Inhume method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 5a66db80c5 [#188] engine: Refactor shard evacuation 
Resolve funlen and gocognit linter for StorageEngine.Evacuate method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 456bc097f7 [#188] engine: Refactor get range from engine 
Resolve funlen linter for StorageEngine.getRange method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 3010ca2649 [#188] engine: Refactor get object from engine 
Resolve funlen linter for StorageEngine.get method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 0739c36a3b [#188] metabase: Refactor object put to metabase 
Resolve funlen linter for db.put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 8273a3dfb2 [#188] blobstor: Refactor blobstor test 
Resolve funlen linter for TestIterate function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 594b5821ed [#188] blobstor: Refactor put data to blobovniczas 
Resolve funlen linter for Blobovniczas.Put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 8e5a0dcf27 [#204] gc: Fix GC handlers start
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 06:48:27 +00:00