TrueCloudLab
/
frostfs-node
19
1
Fork 21
Code Issues 86 Pull Requests 7 Actions Releases Activity
3,861 Commits
7 Branches
21 Tags
47 MiB
Commit Graph

818 Commits (d212d908b5303b56e8b7eabe82560b4a8f58cf4c)

Author SHA1 Message Date
Dmitrii Stepanov d212d908b5 [#312] wc: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Dmitrii Stepanov 4503a61997 [#312] wc: Delete unused Iterate method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Evgenii Stratonikov 4b768fd115 [#381] *: Move to sync/atomic 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-23 08:18:01 +03:00
Evgenii Stratonikov ff570847a4 [#381] go.mod: Update bbolt 
Adopt new `ForEachBucket` function where possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-22 11:49:14 +03:00
Evgenii Stratonikov 35c9b6b26d [#314] writecache: remove objects right after they are flushed
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy bf79d06f03 [#314] writecache: Do not lose small objects on disk errors 
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy 9e56592be3 [#314] writecache: Simplify background workers naming 
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-19 09:37:05 +00:00
Dmitrii Stepanov e4889e06ba [#329] node: Make evacuate async 
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-19 08:43:52 +00:00
Dmitrii Stepanov 869fcbf591 [#332] gc: Fix expired complex object deletion
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-16 12:44:57 +00:00
Dmitrii Stepanov ab07bad33d [#332] gc: Add complex object unit test 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-16 12:44:57 +00:00
Evgenii Stratonikov c62025c836 [#321] metabase/test: execute tests in parallel
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov 945454f60c [#321] engine/test: Execute tests in parallel 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov 4578d00619 [#321] shard/test: Execute tests in parallel 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov d35e4c389f [#321] shard/test: Parallelize TestWriteCacheObjectLoss 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov 969bfb603f [#321] shard/test: Parallelize TestShard_List 
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After:  0.109s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Dmitrii Stepanov a181c9e434 [#332] gc: Add additional logging
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-10 17:34:40 +03:00
Alejandro Lopez 973af12854 [#327] tests: replace os.MkdirTemp with t.TempDir
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-05 11:24:51 +00:00
Evgenii Stratonikov a1b4ba9980 [#209] compression: Do not store uncompressible data 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-03 16:28:29 +03:00
Dmitrii Stepanov eca5c210dd [#299] evacuate: Add context cancel checks
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-02 15:55:30 +03:00
Evgenii Stratonikov a9c4ba62c3 [#248] metabase: Remove storage group bucket 
Backwards compatible change, so no version increase.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 17:00:43 +03:00
Evgenii Stratonikov 58f1ba4b51 [#288] pilorama: Add missing operation in log
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov daa26f6e9b [#288] pilorama/test: Check operation order for TreeGetByPath() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov 291f9e809a [#288] pilorama: Remove getMeta() wrapper 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov 0045f1bcd4 [#288] pilorama: Use more descriptive names for memory tree 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov f856ad7480 [#288] pilorama: Remove `childMap` from memory forest 
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Airat Arifullin ada081dfd5 [#19] node: Make policier read shards concurrently
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Introduce ListWithMultiCursor that simultaneously reads objects
  from different shards

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-27 11:11:47 +03:00
Evgenii Stratonikov 1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo()
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:11 +03:00
Evgenii Stratonikov f07e2d4812 [#285] lint: Fix revive/unused-parameter 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:04 +03:00
Pavel Karpy 09938a9841 Revert "[#262] meta: Do not return old expired objects"
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
This reverts commit 3d23b087

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:55 +03:00
Pavel Karpy e9461686b8 [#274] wc: Resolve possible deadlock 
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Pavel Karpy 6b6f33ed71 [#274] wc: Make wait groups work more explicit 
Do not run routine that calls `wg.Done()` inside, it is hard to read.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Airat Arifullin 6c90bb87f1 [#118] node: add ctx for unit tests for blobstor
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-19 18:09:33 +03:00
Pavel Karpy 3d23b08773 [#262] meta: Do not return old expired objects
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-19 13:37:38 +00:00
Airat Arifullin 13c8afcb02 [#118] node: add unit concurrent tests for blobstor
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-19 10:22:50 +00:00
Pavel Karpy 20cd080323 [#255] write-cache: Fix init race condition
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Do not use WC's internals in the initialization routines without mode
protection. WC should be able to change its mode even if the initialization
is not finished yet.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-19 09:29:07 +00:00
Pavel Karpy 262c9c2b93 [#256] blobovniczaTree: Make `Exists` test stable
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Corrupt and request _the same_ file.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-14 16:46:24 +03:00
Evgenii Stratonikov 8466894fdf [#250] control: remove `DumpShard` and `RestoreShard` RPC
ci/woodpecker/push/pre-commit Pipeline was successful Details 
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 12:28:49 +00:00
Dmitrii Stepanov 6121b541b5 [#242] treesvc: Add tracing spans
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Dmitrii Stepanov d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Alejandro Lopez 41eb3129ae [#139] Refactor blobovnicza exist test to not use chmod
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-04-14 12:16:14 +03:00
Evgenii Stratonikov 0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Evgenii Stratonikov 5d2affa5cd testutil: Fix linter warning 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-12 11:00:02 +03:00
Dmitrii Stepanov 0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Evgenii Stratonikov c4865783fc [#236] blobstor/test: Prefill storage in parallel in read benchmark
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:11 +03:00
Evgenii Stratonikov 6ad5c38225 [#236] testutil: Use random object id in `RandObjGenerator` 
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8            227425              4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8     --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8            --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:04 +03:00
Evgenii Stratonikov c85a0bc866 [#236] blobstor/test: Reduce test descriptions 
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:04 +03:00
Evgenii Stratonikov dbc3811ff4 [#191] engine: Allow to remove redundant object copies 
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-07 17:25:50 +00:00
Airat Arifullin cb172e73a6 [#228] node: Use uber atomic package instead standard
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 15:37:27 +00:00
Dmitrii Stepanov f32f61df87 [#203] pilorama: Refactor tests 
Do not pass 0 as channel capacity.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov 8908798f59 [#203] node: Resolve unused vars 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00