TrueCloudLab/frostfs-node
20
1
Fork 27
Code Issues 92 Pull requests 4 Projects Releases 1 Activity Actions
4851 commits 8 branches 85 tags 54 MiB
Commit graph

386 commits

Author SHA1 Message Date
Dmitrii Stepanov
44552a849b [#661] shard: Add blobstor rebuilder 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov
d30ab5f29e [#838] metabase: Count user objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov
f314da4af3 [#838] metabase: Add user object type counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov
29550fe600 [#838] shard: Refactor updateMetrics method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Alexander Chuprov
445ebcc0e7 [#651] shard/test: Speedup Shard_Delete 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-30 13:19:43 +00:00
Alexander Chuprov
2302e5d342 [#651] shard/test: Refactor Shard_Delete 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-30 13:19:43 +00:00
Dmitrii Stepanov
07390ad4e3 [#715] node: Unify config parameter names
All checks were successful
DCO action / DCO (pull_request) Successful in 2m54s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m11s
Details
Tests and linters / Staticcheck (pull_request) Successful in 3m55s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m51s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m6s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m10s
Details
Tests and linters / Lint (pull_request) Successful in 5m24s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m56s
Details
Tests and linters / Tests with -race (pull_request) Successful in 7m46s
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-22 17:13:50 +03:00
Dmitrii Stepanov
76ff26039c [#96] node: Drop neo-go's slices package
All checks were successful
DCO action / DCO (pull_request) Successful in 2m13s
Details
Build / Build Components (1.20) (pull_request) Successful in 5m48s
Details
Build / Build Components (1.21) (pull_request) Successful in 5m47s
Details
Vulncheck / Vulncheck (pull_request) Successful in 5m47s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m33s
Details
Tests and linters / Staticcheck (pull_request) Successful in 7m37s
Details
Tests and linters / Lint (pull_request) Successful in 8m8s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 10m38s
Details
Tests and linters / Tests with -race (pull_request) Successful in 11m22s
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-17 13:24:04 +03:00
Dmitrii Stepanov
70ab1ebd54 [#763] metrics: Add container_objects_total metric
All checks were successful
DCO action / DCO (pull_request) Successful in 3m54s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Details
Build / Build Components (1.21) (pull_request) Successful in 5m16s
Details
Vulncheck / Vulncheck (pull_request) Successful in 9m54s
Details
Tests and linters / Lint (pull_request) Successful in 10m57s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 12m40s
Details
Tests and linters / Staticcheck (pull_request) Successful in 12m34s
Details
Tests and linters / Tests with -race (pull_request) Successful in 12m48s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 13m19s
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov
79088baa06 [#772] node: Apply gofumpt 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:03 +03:00
Evgenii Stratonikov
00aa6d9749 [#633] shard/test: Fix TestCounters() 
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Dmitrii Stepanov
f2437f7ae9 [#734] shard: Fix Delete method
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Details
Build / Build Components (1.20) (pull_request) Successful in 3m46s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m16s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m42s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m35s
Details
Tests and linters / Tests with -race (pull_request) Successful in 6m33s
Details
Build / Build Components (1.21) (pull_request) Successful in 13m7s
Details
Tests and linters / Lint (pull_request) Successful in 19m14s
Details 
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Alexander Chuprov
c1e4130020 [#146] node: Add trace_id to logs
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
Details
DCO action / DCO (pull_request) Successful in 3m36s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Details
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Details
Tests and linters / Lint (pull_request) Successful in 5m2s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Details
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Details 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-09-27 11:05:27 +03:00
Ekaterina Lebedeva
8a81af5a3b [#653] Add context parameter to Open functions
All checks were successful
DCO action / DCO (pull_request) Successful in 1m38s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m22s
Details
Build / Build Components (1.21) (pull_request) Successful in 4m25s
Details
Vulncheck / Vulncheck (pull_request) Successful in 4m56s
Details
Tests and linters / Lint (pull_request) Successful in 6m1s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s
Details
Tests and linters / Staticcheck (pull_request) Successful in 8m1s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s
Details
Tests and linters / Tests with -race (pull_request) Successful in 8m32s
Details 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-09-07 18:03:29 +03:00
Evgenii Stratonikov
a716db99db [#668] shard/test: Do not alter rootPath option 
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
36759f8434 [#668] shard/test: Properly check event processing 
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
39879fa868 [#668] shard/test: Add dontRelease options 
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
268adb79cb [#668] shard/test: Simplify shard construction 
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
429f941cda [#668] shard/test: Release shard in t.Cleanup() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
382eb8a485 [#668] shard/test: Disable GC where it is not needed 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
42696016de [#668] shard: Close stopChannel in GC 
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
bdecfbc1be [#668] shard/test: Move tests to the main package 
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters 
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
da8f384324 [#668] shard/test: Fix typo in existence 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-31 08:39:42 +00:00
Evgenii Stratonikov
9072772a09 [#649] shard/test: Increase GC remover interval 
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:10:25 +00:00
Evgenii Stratonikov
c4db8e7690 [#637] shard/test: Fix data race 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:10:25 +00:00
Evgenii Stratonikov
96e690883f [#638] Unify test loggers 
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
322c1dc273 [#638] Use test.NewLogger() in tests 
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"

-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
82cc453be9 [#xx] shard: Fix data race in metrics tests 
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 10:26:12 +00:00
Alexander Chuprov
4d9a6c07fb [#618] core: Replace fmt.Sprintf with strconv.FormatUint 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-08-21 07:14:50 +00:00
Evgenii Stratonikov
f7042c5a6f [#609] Replace zaptest.NewLogger() with zap.L() 
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"

-zaptest.NewLogger(t)
+zap.L()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:51:10 +00:00
Evgenii Stratonikov
e604a3d749 [#607] *: Use zap.Stringer() where possible 
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
 f(
    ...,
-    zap.String(t, a.String()),
+    zap.Stringer(t, a),
    ...,
)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:50:02 +00:00
Alexander Chuprov
21800e9fcc [#162] core: Move literals to constants 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-08-11 15:48:42 +03:00
Evgenii Stratonikov
d641cba2fc [#587] Do not use math/rand.Read
Some checks failed
Build / Build Components (1.20) (pull_request) Failing after 3s
Details
Build / Build Components (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Lint (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.20) (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Tests with -race (pull_request) Failing after 5s
Details
Tests and linters / Staticcheck (pull_request) Failing after 8s
Details
Vulncheck / Vulncheck (pull_request) Failing after 2s
Details 
Fix staticcheck warnings after go1.20 update.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-09 16:02:44 +03:00
Alejandro Lopez
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 10:26:53 +00:00
Alejandro Lopez
1a0cb0f34a [#421] Try using badger for the write-cache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-07 08:16:57 +00:00
Dmitrii Stepanov
b4e72a2dfd [#335] treesvc: Sort nodes by Filename in GetSubTree 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-07-20 10:14:10 +03:00
Alexander Chuprov
033eaf77e1 [#496] node: Fix linter importas
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Details
Build / Build Components (1.19) (pull_request) Successful in 4m1s
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Details
Tests and linters / Lint (pull_request) Successful in 14m40s
Details
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-07-06 15:36:41 +03:00
Dmitrii Stepanov
cac4ed93d6 [#428] engine: Add low_mem config parameter 
Concurrent initialization in case of the metabase resync leads to
high memory consumption and potential OOM.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-26 13:29:39 +00:00
Dmitrii Stepanov
16a142cd0c [#373] metrics: Add FSTree metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
d8ecc69d00 [#373] local storage: Pass parent ID 
This is required to add shard ID as metric label.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
059e9e88a2 [#373] metabase: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
01a0c97760 [#453] engine: Set Disabled mode to deleted shard
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-20 12:04:07 +03:00
Dmitrii Stepanov
4449006862 [#424] metrics: Use mode value as metric value for shard 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 18:26:19 +03:00
Dmitrii Stepanov
1b364d8cf4 [#424] metrics: Refactor engine metrics 
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:53:32 +03:00
Dmitrii Stepanov
2541d319de [#266] pilorama: Allow to get current tree height 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-13 10:00:45 +00:00
Anton Nikiforov
263c6fdc50 [#372] node: Add metrics for the error counter in the engine
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-06-07 13:04:47 +00:00
Dmitrii Stepanov
74578052f9 [#412] node: Replace tracing package 
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Dmitrii Stepanov
3220c4df9f [#376] metrics: Add GC metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-31 10:22:12 +00:00
Anton Nikiforov
365a7ca0f4 [#366] node: Stop GC once termination signal received
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-29 09:35:08 +03:00
Anton Nikiforov
802168c0c6 [#364] node: Stop flushing big object when termination signal received
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-26 16:46:58 +03:00
Dmitrii Stepanov
20a489bdb5 [#393] gc: Use defer to mark handler done 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-26 12:14:02 +00:00
Dmitrii Stepanov
2613351008 [#387] gc: Cancel GC is change mode requested
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-25 09:38:16 +03:00
Dmitrii Stepanov
2ce43935f9 [#312] metrics: Add writecache metrcis 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Evgenii Stratonikov
4b768fd115 [#381] *: Move to sync/atomic 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-23 08:18:01 +03:00
Dmitrii Stepanov
e4889e06ba [#329] node: Make evacuate async 
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-19 08:43:52 +00:00
Dmitrii Stepanov
869fcbf591 [#332] gc: Fix expired complex object deletion
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-16 12:44:57 +00:00
Dmitrii Stepanov
ab07bad33d [#332] gc: Add complex object unit test 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-16 12:44:57 +00:00
Evgenii Stratonikov
4578d00619 [#321] shard/test: Execute tests in parallel 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
d35e4c389f [#321] shard/test: Parallelize TestWriteCacheObjectLoss 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
969bfb603f [#321] shard/test: Parallelize TestShard_List 
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After:  0.109s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Dmitrii Stepanov
a181c9e434 [#332] gc: Add additional logging
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-10 17:34:40 +03:00
Evgenii Stratonikov
8466894fdf [#250] control: remove DumpShard and RestoreShard RPC
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 12:28:49 +00:00
Dmitrii Stepanov
6121b541b5 [#242] treesvc: Add tracing spans
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Dmitrii Stepanov
d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Evgenii Stratonikov
0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Dmitrii Stepanov
0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Airat Arifullin
cb172e73a6 [#228] node: Use uber atomic package instead standard
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 15:37:27 +00:00
Dmitrii Stepanov
8908798f59 [#203] node: Resolve unused vars 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
9e2df4b7c7 [#203] node: Fix double imports 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
23575e1ac0 [#210] policier: Resolve contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Evgenii Stratonikov
9098d0eec0 [#211] engine: Unify shard mode checks for tree operations
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-05 11:10:39 +00:00
Dmitrii Stepanov
8e5a0dcf27 [#204] gc: Fix GC handlers start
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 06:48:27 +00:00
Dmitrii Stepanov
a7c79c773a [#168] node: Refactor node config 
Resolve containedctx linter for cfg

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-31 09:32:59 +03:00
Dmitrii Stepanov
9f0bce5c15 [#183] gc: Fix drop expired locked simple objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 15:33:42 +03:00
Alejandro Lopez
341fe1688f [#139] test: Add test storage implementation
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-03-29 14:28:49 +00:00
aarifullin
34329d67ff [#86] node: Fix unit test and linter errors
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
 2023-03-23 12:42:58 +03:00
Ayrat Arifullin
9808dec591 [#86] node: Move testing utils to one package
Some checks failed
ci/woodpecker/push/pre-commit Pipeline failed
Details 
Move testing utils from tests in local_object_storage package to
unified testutil package

Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
 2023-03-23 08:19:15 +00:00
Evgenii Stratonikov
fb13902db9 [#156] shard: Make refillMetabase() pass linter checks 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov
47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply() 
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Dmitrii Stepanov
5059dcc19d [#145] shard-gc: Delete expired objects after locks 
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 11:31:08 +03:00
Dmitrii Stepanov
6c4a1699ef [#145] shard-gc: Expired locked unit test 
Added unit test that verifies that GC deletes expired
locked objects in one epoch.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 11:31:08 +03:00
Dmitrii Stepanov
97c36ed3ec [#148] linter: Add funlen linter 
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Evgenii Stratonikov
3e6fd4c611 [#82] pilorama: Allow to store last sync height 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 11:25:44 +00:00
Alex Vanin
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Pavel Karpy
3beef10f89 [#61] node: Do not fetch missing objects 
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 14:47:38 +03:00
Pavel Karpy
07ec51ea60 [#2244] node: Add object address to WC's operations 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
427fe276f2 [#2238] shard: Try closing all components 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Artem Tataurov
362f24953a [#47] shard: Switch container size metric from physical to logical capacity 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-02-17 12:03:42 +03:00
Artem Tataurov
ab21d90cfb [#1794] shard: Add increasing case for the payload size metric 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-02-09 13:30:23 +03:00
Stanislav Bogatyrev
cb016d53a6 [#1] Fix comments and error messages 
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
 2023-02-06 17:41:14 +03:00
Pavel Karpy
89a0266f5e [#1794] metrics: Track physical object capacity per shard 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-01-26 20:06:28 +03:00
Evgenii Stratonikov
9513f163aa [#2116] metrics: Track physical object capacity in the container 
Currently we track based on `PayloadSize`, because it is already stored
in the metabase and it is easier to calculate without slowing down the
whole system.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-01-26 20:06:28 +03:00
Evgenii Stratonikov
d65a95a2c6 [#28] pilorama: Remove LogMove struct 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
c72576e72f [#2208] engine: Log time-consuming shard operations 
Currently the only way to tell whether `evacuate/set-mode` is finished
is to set a very big timeout and _hope_ that the operation will finish.
In this commit we add INFO logs for such operations which should
simplify the life of an administrator.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Pavel Karpy
64a5294b27 [#2200] shard: Do not fetch big objects from blobovniczas 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Pavel Karpy
91757329ae [#2200] shard: Fix blobstor obj fetching 
In the previous implementation any non-nil error that preceded object
fetching from blobstor led to iterating over every storage (in other words,
no storage ID information was taken into account). Now storage ID is
skipped only if metabase (storage ID source) returns any error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00