TrueCloudLab/frostfs-node
21
1
Fork 27
Code Issues 104 Pull requests 4 Projects Releases 2 Activity Actions
3917 commits 9 branches 96 tags 51 MiB
Commit graph

834 commits

Author SHA1 Message Date
Dmitrii Stepanov
07f155ac77 [#424] metrics: Use labels for writecache methods and operations 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:52:40 +03:00
Dmitrii Stepanov
2541d319de [#266] pilorama: Allow to get current tree height 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-13 10:00:45 +00:00
Evgenii Stratonikov
41ab4d070e [#423] *: Use hrw.StringHash() where possible
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-13 07:18:25 +00:00
Anton Nikiforov
263c6fdc50 [#372] node: Add metrics for the error counter in the engine
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-06-07 13:04:47 +00:00
Dmitrii Stepanov
74578052f9 [#412] node: Replace tracing package 
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Dmitrii Stepanov
dbf41391b5 [#401] engine: Extend evacuation logs
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Add operation-tag to logger.
Log evacuation results.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-31 13:24:30 +03:00
Dmitrii Stepanov
3220c4df9f [#376] metrics: Add GC metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-31 10:22:12 +00:00
Alejandro Lopez
faca861451 [#411] Remove unnecessary pointers for sync objects
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-31 10:19:14 +00:00
Anton Nikiforov
8dcd06c587 [#394] node: Use Context in Blobovniczas.Iterate()
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-31 10:09:18 +00:00
Anton Nikiforov
365a7ca0f4 [#366] node: Stop GC once termination signal received
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-29 09:35:08 +03:00
Anton Nikiforov
802168c0c6 [#364] node: Stop flushing big object when termination signal received
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-26 16:46:58 +03:00
Dmitrii Stepanov
271a56c2ab [#395] metrics: Drop redundant metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
HistogramVec already has labeled counter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-26 13:39:37 +00:00
Evgenii Stratonikov
f2e5dead7e [#398] pilorama: Disallow applying same operations
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
   overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-26 12:14:29 +00:00
Dmitrii Stepanov
20a489bdb5 [#393] gc: Use defer to mark handler done 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-26 12:14:02 +00:00
Dmitrii Stepanov
2613351008 [#387] gc: Cancel GC is change mode requested
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-25 09:38:16 +03:00
Dmitrii Stepanov
2ce43935f9 [#312] metrics: Add writecache metrcis 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Dmitrii Stepanov
d212d908b5 [#312] wc: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Dmitrii Stepanov
4503a61997 [#312] wc: Delete unused Iterate method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Evgenii Stratonikov
4b768fd115 [#381] *: Move to sync/atomic 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-23 08:18:01 +03:00
Evgenii Stratonikov
ff570847a4 [#381] go.mod: Update bbolt 
Adopt new `ForEachBucket` function where possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-22 11:49:14 +03:00
Evgenii Stratonikov
35c9b6b26d [#314] writecache: remove objects right after they are flushed
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy
bf79d06f03 [#314] writecache: Do not lose small objects on disk errors 
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy
9e56592be3 [#314] writecache: Simplify background workers naming 
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-19 09:37:05 +00:00
Dmitrii Stepanov
e4889e06ba [#329] node: Make evacuate async 
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-19 08:43:52 +00:00
Dmitrii Stepanov
869fcbf591 [#332] gc: Fix expired complex object deletion
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-16 12:44:57 +00:00
Dmitrii Stepanov
ab07bad33d [#332] gc: Add complex object unit test 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-16 12:44:57 +00:00
Evgenii Stratonikov
c62025c836 [#321] metabase/test: execute tests in parallel
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
945454f60c [#321] engine/test: Execute tests in parallel 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
4578d00619 [#321] shard/test: Execute tests in parallel 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
d35e4c389f [#321] shard/test: Parallelize TestWriteCacheObjectLoss 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
969bfb603f [#321] shard/test: Parallelize TestShard_List 
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After:  0.109s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Dmitrii Stepanov
a181c9e434 [#332] gc: Add additional logging
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-10 17:34:40 +03:00
Alejandro Lopez
973af12854 [#327] tests: replace os.MkdirTemp with t.TempDir
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-05 11:24:51 +00:00
Evgenii Stratonikov
a1b4ba9980 [#209] compression: Do not store uncompressible data 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-03 16:28:29 +03:00
Dmitrii Stepanov
eca5c210dd [#299] evacuate: Add context cancel checks
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-02 15:55:30 +03:00
Evgenii Stratonikov
a9c4ba62c3 [#248] metabase: Remove storage group bucket 
Backwards compatible change, so no version increase.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 17:00:43 +03:00
Evgenii Stratonikov
58f1ba4b51 [#288] pilorama: Add missing operation in log
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
daa26f6e9b [#288] pilorama/test: Check operation order for TreeGetByPath() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
291f9e809a [#288] pilorama: Remove getMeta() wrapper 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
0045f1bcd4 [#288] pilorama: Use more descriptive names for memory tree 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
f856ad7480 [#288] pilorama: Remove childMap from memory forest 
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Airat Arifullin
ada081dfd5 [#19] node: Make policier read shards concurrently
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
* Introduce ListWithMultiCursor that simultaneously reads objects
  from different shards

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-27 11:11:47 +03:00
Evgenii Stratonikov
1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo()
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:11 +03:00
Evgenii Stratonikov
f07e2d4812 [#285] lint: Fix revive/unused-parameter 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:04 +03:00
Pavel Karpy
09938a9841 Revert "[#262] meta: Do not return old expired objects"
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
This reverts commit 3d23b087

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:55 +03:00
Pavel Karpy
e9461686b8 [#274] wc: Resolve possible deadlock 
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Pavel Karpy
6b6f33ed71 [#274] wc: Make wait groups work more explicit 
Do not run routine that calls `wg.Done()` inside, it is hard to read.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Airat Arifullin
6c90bb87f1 [#118] node: add ctx for unit tests for blobstor
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-19 18:09:33 +03:00
Pavel Karpy
3d23b08773 [#262] meta: Do not return old expired objects
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-19 13:37:38 +00:00
Airat Arifullin
13c8afcb02 [#118] node: add unit concurrent tests for blobstor
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-19 10:22:50 +00:00