c6a739e746
[ #661 ] blobovniczatree: Make Rebuild concurrent for objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
f1c7905263
[ #661 ] blobovniczatree: Make Rebuild concurrent
...
Different DBs can be rebuild concurrently.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
d4d905ecc6
[ #661 ] metrics: Add blobovniczatree rebuild metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
b2769ca3de
[ #661 ] blobovniczatree: Make Rebuild failover safe
...
Now move info stores in blobovnicza, so in case of failover
rebuild completes previous operation first.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
da4fee2d0b
[ #698 ] blobovniczatree: Init blobovniczas concurrently
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
422226da18
[ #661 ] blobovniczatree: Add Rebuild implementation
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
a531eaf8bc
[ #661 ] blobstor: Add Rebuild implementation
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
c1667a11d2
[ #661 ] blobovniczatree: Allow to change depth or width
...
Now it is possible to change depth or with of blobovniczatree.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
484eb59893
[ #661 ] blobovniczatree: Use .db extension for db files
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
44552a849b
[ #661 ] shard: Add blobstor rebuilder
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
76ff26039c
[ #96 ] node: Drop neo-go's slices package
...
DCO action / DCO (pull_request) Successful in 2m13s
Build / Build Components (1.20) (pull_request) Successful in 5m48s
Build / Build Components (1.21) (pull_request) Successful in 5m47s
Vulncheck / Vulncheck (pull_request) Successful in 5m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m33s
Tests and linters / Staticcheck (pull_request) Successful in 7m37s
Tests and linters / Lint (pull_request) Successful in 8m8s
Tests and linters / Tests (1.20) (pull_request) Successful in 10m38s
Tests and linters / Tests with -race (pull_request) Successful in 11m22s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 13:24:04 +03:00
29fe8c41f3
[ #655 ] storage: Drop ErrorHandler
...
DCO action / DCO (pull_request) Successful in 1m42s
Build / Build Components (1.21) (pull_request) Successful in 3m23s
Vulncheck / Vulncheck (pull_request) Successful in 3m25s
Build / Build Components (1.20) (pull_request) Successful in 4m16s
Tests and linters / Staticcheck (pull_request) Successful in 4m24s
Tests and linters / Lint (pull_request) Successful in 5m2s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m36s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m42s
Tests and linters / Tests with -race (pull_request) Successful in 7m18s
The only one usage was for logging.
Now logging performed by storage anyway.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
137e987a4e
[ #655 ] storage: Drop LazyHandler
...
LazyHandler is implemented and used incorrectly.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
c80b46fad3
[ #754 ] blobstor: Estimate compressability
...
DCO action / DCO (pull_request) Successful in 1m59s
Vulncheck / Vulncheck (pull_request) Successful in 3m31s
Build / Build Components (1.20) (pull_request) Successful in 4m37s
Build / Build Components (1.21) (pull_request) Successful in 4m33s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m54s
Tests and linters / Staticcheck (pull_request) Successful in 4m49s
Tests and linters / Tests with -race (pull_request) Successful in 5m9s
Tests and linters / Lint (pull_request) Successful in 6m4s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m9s
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 11:24:32 +03:00
79088baa06
[ #772 ] node: Apply gofumpt
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
c1e4130020
[ #146 ] node: Add trace_id to logs
...
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
DCO action / DCO (pull_request) Successful in 3m36s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Tests and linters / Lint (pull_request) Successful in 5m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
8a81af5a3b
[ #653 ] Add context parameter to Open functions
...
DCO action / DCO (pull_request) Successful in 1m38s
Build / Build Components (1.20) (pull_request) Successful in 4m22s
Build / Build Components (1.21) (pull_request) Successful in 4m25s
Vulncheck / Vulncheck (pull_request) Successful in 4m56s
Tests and linters / Lint (pull_request) Successful in 6m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s
Tests and linters / Staticcheck (pull_request) Successful in 8m1s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s
Tests and linters / Tests with -race (pull_request) Successful in 8m32s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
7456c8556a
[ #536 ] blobovnicza: Add blobovniczatree DB cache
...
DCO action / DCO (pull_request) Successful in 1m33s
Vulncheck / Vulncheck (pull_request) Successful in 2m43s
Build / Build Components (1.20) (pull_request) Successful in 3m40s
Tests and linters / Staticcheck (pull_request) Successful in 3m59s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m3s
Tests and linters / Tests with -race (pull_request) Successful in 6m28s
Build / Build Components (1.21) (pull_request) Successful in 12m34s
Tests and linters / Lint (pull_request) Successful in 12m51s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:53:11 +03:00
c672f59ab8
[ #536 ] blobovnicza: Drop cache
...
Each blobovnicza instance is opened
while is in use.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:51:26 +03:00
96e690883f
[ #638 ] Unify test loggers
...
In some places we have debug=false, in others debug=true.
Let's be consistent.
Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
322c1dc273
[ #638 ] Use test.NewLogger() in tests
...
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"
-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
354a92ea2c
[ #602 ] blobovnicza: Add leaf width implementation
...
Vulncheck / Vulncheck (pull_request) Successful in 1m14s
Build / Build Components (1.21) (pull_request) Successful in 3m36s
DCO action / DCO (pull_request) Successful in 4m4s
Tests and linters / Staticcheck (pull_request) Successful in 4m5s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m49s
Tests and linters / Lint (pull_request) Successful in 5m18s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m44s
Tests and linters / Tests with -race (pull_request) Successful in 5m47s
Build / Build Components (1.20) (pull_request) Successful in 10m45s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
c4e1d8eb07
[ #602 ] node: Fix blobovnicza typos
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 11:14:10 +03:00
2e49d7ea7e
[ #602 ] blobovnicza: Init before using
...
Fix blobovnicza size: after restart size metric resets.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
f7042c5a6f
[ #609 ] Replace zaptest.NewLogger() with zap.L()
...
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"
-zaptest.NewLogger(t)
+zap.L()
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:51:10 +00:00
a8de37c8a2
[ #607 ] *: Remove redundant if on error returns
...
Semantic patch:
```
@@
@@
-if err != nil { return err }
-return nil
+return err
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
58c8722c81
[ #585 ] fstree: Add optional file counter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
baad49990c
[ #585 ] fstree: Return logical error if object deleted
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
0c52186572
[ #585 ] fstree: Remove unused method
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
ae8be495c8
[#xx] Avoid manual management of files in tests
...
Vulncheck / Vulncheck (pull_request) Successful in 2m36s
Build / Build Components (1.21) (pull_request) Successful in 3m33s
DCO action / DCO (pull_request) Successful in 3m33s
Tests and linters / Staticcheck (pull_request) Successful in 4m15s
Tests and linters / Lint (pull_request) Successful in 5m31s
Build / Build Components (1.20) (pull_request) Successful in 7m2s
Tests and linters / Tests with -race (pull_request) Failing after 8m51s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m55s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m6s
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 14:01:39 +03:00
b44a8dd46c
[ #597 ] *: Fix linter warnings
...
Vulncheck / Vulncheck (pull_request) Successful in 6m26s
Tests and linters / Staticcheck (pull_request) Successful in 7m20s
Build / Build Components (1.21) (pull_request) Successful in 8m31s
Build / Build Components (1.20) (pull_request) Successful in 8m43s
Tests and linters / Tests (1.21) (pull_request) Failing after 15m48s
Tests and linters / Tests (1.20) (pull_request) Failing after 16m2s
Tests and linters / Tests with -race (pull_request) Failing after 16m28s
Tests and linters / Lint (pull_request) Successful in 19m8s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:55:51 +03:00
8f994163ee
[ #586 ] Fix writecache benchmarks and refactor hacky NeedsCompression
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 08:05:18 +00:00
d641cba2fc
[ #587 ] Do not use math/rand.Read
...
Build / Build Components (1.20) (pull_request) Failing after 3s
Build / Build Components (1.21) (pull_request) Failing after 3s
Tests and linters / Lint (pull_request) Failing after 3s
Tests and linters / Tests (1.20) (pull_request) Failing after 3s
Tests and linters / Tests (1.21) (pull_request) Failing after 3s
Tests and linters / Tests with -race (pull_request) Failing after 5s
Tests and linters / Staticcheck (pull_request) Failing after 8s
Vulncheck / Vulncheck (pull_request) Failing after 2s
Fix staticcheck warnings after go1.20 update.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
5b7e4a51b7
[ #481 ] Update frostfs-sdk-go and error pointer receivers
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 10:26:53 +00:00
65c72f3e0b
[ #559 ] Remove manual path handling in fstree tests
...
Build / Build Components (1.19) (pull_request) Successful in 2m8s
Build / Build Components (1.20) (pull_request) Successful in 1m54s
Tests and linters / Lint (pull_request) Successful in 2m48s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m53s
Tests and linters / Tests (1.20) (pull_request) Successful in 2m4s
Tests and linters / Tests with -race (pull_request) Successful in 5m21s
Tests and linters / Staticcheck (pull_request) Successful in 2m16s
Vulncheck / Vulncheck (pull_request) Failing after 16m3s
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-03 10:03:41 +03:00
4d48377cec
[ #459 ] blobovniczatree: Fix get error
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 16:44:53 +03:00
af608da952
[ #373 ] metrics: Add blobovnizca metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d8ecc69d00
[ #373 ] local storage: Pass parent ID
...
This is required to add shard ID as metric label.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
f54cc0b607
[ #373 ] blobstor: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
8318d90ad0
[ #373 ] blobovniczatree: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
3ae3c8dfdb
[ #373 ] fstree: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
a8526d45e9
[ #373 ] blobovnizca: Add missed/fix tracing spans
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
41ab4d070e
[ #423 ] *: Use hrw.StringHash() where possible
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-13 07:18:25 +00:00
74578052f9
[ #412 ] node: Replace tracing package
...
Use observability module.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
8dcd06c587
[ #394 ] node: Use Context in Blobovniczas.Iterate()
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-31 10:09:18 +00:00
802168c0c6
[ #364 ] node: Stop flushing big object when termination signal received
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
973af12854
[ #327 ] tests: replace os.MkdirTemp with t.TempDir
...
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-05 11:24:51 +00:00
a1b4ba9980
[ #209 ] compression: Do not store uncompressible data
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
f07e2d4812
[ #285 ] lint: Fix revive/unused-parameter
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
6c90bb87f1
[ #118 ] node: add ctx for unit tests for blobstor
...
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 18:09:33 +03:00
