Commit graph

353 commits

Author SHA1 Message Date
e3573de6db [#930] gc: Stop internal activity by context
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
675eec91f3 [#938] shard: Update only changed counters
All checks were successful
DCO action / DCO (pull_request) Successful in 5m40s
Vulncheck / Vulncheck (pull_request) Successful in 5m33s
Tests and linters / Staticcheck (pull_request) Successful in 7m12s
Build / Build Components (1.21) (pull_request) Successful in 7m25s
Build / Build Components (1.20) (pull_request) Successful in 7m29s
Tests and linters / Lint (pull_request) Successful in 8m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m48s
Tests and linters / Tests with -race (pull_request) Successful in 9m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m10s
If metric value hasn't changed, but we update metric, then
non existed metric will apear with zero value.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
931a5e9aaf [#918] engine: Move shard to degraded mode if metabase open failed
All checks were successful
DCO action / DCO (pull_request) Successful in 1m55s
Build / Build Components (1.20) (pull_request) Successful in 2m14s
Vulncheck / Vulncheck (pull_request) Successful in 2m15s
Build / Build Components (1.21) (pull_request) Successful in 4m6s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m58s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m24s
Tests and linters / Staticcheck (pull_request) Successful in 6m11s
Tests and linters / Lint (pull_request) Successful in 6m37s
Tests and linters / Tests with -race (pull_request) Successful in 6m35s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 11:16:40 +03:00
f526f49995 [#874] engine: Check object existance concurrently
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Vulncheck / Vulncheck (pull_request) Successful in 2m33s
Build / Build Components (1.21) (pull_request) Successful in 3m9s
Build / Build Components (1.20) (pull_request) Successful in 4m49s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m5s
Tests and linters / Staticcheck (pull_request) Successful in 6m6s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m39s
Tests and linters / Lint (pull_request) Successful in 7m0s
Tests and linters / Tests with -race (pull_request) Successful in 6m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 09:28:29 +03:00
57171907e3 [#904] metabase: Return if object was actuall inserted
This requires to count metrics properly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
a2ab373a0a [#895] metabase: Do not delete GC mark for virtual objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
47dcfa20f3 [#895] test: Use t.Cleanup only for external resources
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
f1b2b8bffa [#895] test: Fix NewLogger arguments list
`debug` is always true.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
4b8b4da681 [#864] engine: Drop container count metric if container removed
All checks were successful
DCO action / DCO (pull_request) Successful in 1m30s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m53s
Tests and linters / Lint (pull_request) Successful in 4m31s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m1s
Tests and linters / Staticcheck (pull_request) Successful in 4m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m13s
Tests and linters / Tests with -race (pull_request) Successful in 8m34s
Vulncheck / Vulncheck (pull_request) Successful in 58s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:45:32 +03:00
d75e7e9a21 [#864] engine: Drop container size metric if container deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:44:54 +03:00
dfd62ca6b1 [#864] metabase: Refactor delete/inhume
Available -> Logic, Raw -> Phy for delete/inhume results.
Use single counter instead of vectors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-09 09:59:42 +03:00
581887148a [#569] cli: Add control shards writecache seal command
All checks were successful
DCO action / DCO (pull_request) Successful in 2m46s
Vulncheck / Vulncheck (pull_request) Successful in 3m3s
Build / Build Components (1.21) (pull_request) Successful in 3m53s
Build / Build Components (1.20) (pull_request) Successful in 4m0s
Tests and linters / Staticcheck (pull_request) Successful in 5m31s
Tests and linters / Lint (pull_request) Successful in 6m17s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m14s
Tests and linters / Tests with -race (pull_request) Successful in 13m12s
It does the same as `control shards flush-writecache --seal`, but
has better name.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
0cb0fc1735 [#569] writecache: Allow to seal writecache after flush
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
8180a0664f [#887] node: Drop badger writecache implementation
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 6m1s
Build / Build Components (1.21) (pull_request) Successful in 7m37s
Build / Build Components (1.20) (pull_request) Successful in 7m52s
Tests and linters / Staticcheck (pull_request) Successful in 8m56s
Tests and linters / Lint (pull_request) Successful in 9m26s
Tests and linters / Tests (1.21) (pull_request) Successful in 15m5s
Tests and linters / Tests with -race (pull_request) Successful in 15m7s
DCO action / DCO (pull_request) Successful in 1m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m1s
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-22 13:00:54 +03:00
3b7c0362a8 [#861] shard: Fix Delete object
All checks were successful
DCO action / DCO (pull_request) Successful in 12m22s
Build / Build Components (1.20) (pull_request) Successful in 12m44s
Build / Build Components (1.21) (pull_request) Successful in 13m2s
Vulncheck / Vulncheck (pull_request) Successful in 13m3s
Tests and linters / Staticcheck (pull_request) Successful in 15m32s
Tests and linters / Lint (pull_request) Successful in 16m26s
Tests and linters / Tests (1.21) (pull_request) Successful in 26m11s
Tests and linters / Tests with -race (pull_request) Successful in 28m29s
Tests and linters / Tests (1.20) (pull_request) Successful in 3m28s
It is possible that object doesn't exist in metabase.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 14:25:40 +03:00
f1c7905263 [#661] blobovniczatree: Make Rebuild concurrent
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
422226da18 [#661] blobovniczatree: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
44552a849b [#661] shard: Add blobstor rebuilder
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
d30ab5f29e [#838] metabase: Count user objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
f314da4af3 [#838] metabase: Add user object type counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
29550fe600 [#838] shard: Refactor updateMetrics method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
445ebcc0e7 [#651] shard/test: Speedup Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
2302e5d342 [#651] shard/test: Refactor Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
07390ad4e3 [#715] node: Unify config parameter names
All checks were successful
DCO action / DCO (pull_request) Successful in 2m54s
Vulncheck / Vulncheck (pull_request) Successful in 3m11s
Tests and linters / Staticcheck (pull_request) Successful in 3m55s
Build / Build Components (1.21) (pull_request) Successful in 3m51s
Build / Build Components (1.20) (pull_request) Successful in 4m6s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m10s
Tests and linters / Lint (pull_request) Successful in 5m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m56s
Tests and linters / Tests with -race (pull_request) Successful in 7m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 17:13:50 +03:00
76ff26039c [#96] node: Drop neo-go's slices package
All checks were successful
DCO action / DCO (pull_request) Successful in 2m13s
Build / Build Components (1.20) (pull_request) Successful in 5m48s
Build / Build Components (1.21) (pull_request) Successful in 5m47s
Vulncheck / Vulncheck (pull_request) Successful in 5m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m33s
Tests and linters / Staticcheck (pull_request) Successful in 7m37s
Tests and linters / Lint (pull_request) Successful in 8m8s
Tests and linters / Tests (1.20) (pull_request) Successful in 10m38s
Tests and linters / Tests with -race (pull_request) Successful in 11m22s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 13:24:04 +03:00
70ab1ebd54 [#763] metrics: Add container_objects_total metric
All checks were successful
DCO action / DCO (pull_request) Successful in 3m54s
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Build / Build Components (1.21) (pull_request) Successful in 5m16s
Vulncheck / Vulncheck (pull_request) Successful in 9m54s
Tests and linters / Lint (pull_request) Successful in 10m57s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m40s
Tests and linters / Staticcheck (pull_request) Successful in 12m34s
Tests and linters / Tests with -race (pull_request) Successful in 12m48s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m19s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
79088baa06 [#772] node: Apply gofumpt
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
00aa6d9749 [#633] shard/test: Fix TestCounters()
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
f2437f7ae9 [#734] shard: Fix Delete method
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Build / Build Components (1.20) (pull_request) Successful in 3m46s
Vulncheck / Vulncheck (pull_request) Successful in 3m16s
Tests and linters / Staticcheck (pull_request) Successful in 4m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m35s
Tests and linters / Tests with -race (pull_request) Successful in 6m33s
Build / Build Components (1.21) (pull_request) Successful in 13m7s
Tests and linters / Lint (pull_request) Successful in 19m14s
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
c1e4130020 [#146] node: Add trace_id to logs
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
DCO action / DCO (pull_request) Successful in 3m36s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Tests and linters / Lint (pull_request) Successful in 5m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
8a81af5a3b [#653] Add context parameter to Open functions
All checks were successful
DCO action / DCO (pull_request) Successful in 1m38s
Build / Build Components (1.20) (pull_request) Successful in 4m22s
Build / Build Components (1.21) (pull_request) Successful in 4m25s
Vulncheck / Vulncheck (pull_request) Successful in 4m56s
Tests and linters / Lint (pull_request) Successful in 6m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s
Tests and linters / Staticcheck (pull_request) Successful in 8m1s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s
Tests and linters / Tests with -race (pull_request) Successful in 8m32s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
a716db99db [#668] shard/test: Do not alter rootPath option
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
36759f8434 [#668] shard/test: Properly check event processing
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
39879fa868 [#668] shard/test: Add dontRelease options
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
268adb79cb [#668] shard/test: Simplify shard construction
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
429f941cda [#668] shard/test: Release shard in t.Cleanup()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
382eb8a485 [#668] shard/test: Disable GC where it is not needed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
42696016de [#668] shard: Close stopChannel in GC
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
bdecfbc1be [#668] shard/test: Move tests to the main package
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
da8f384324 [#668] shard/test: Fix typo in existence
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-31 08:39:42 +00:00
9072772a09 [#649] shard/test: Increase GC remover interval
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
c4db8e7690 [#637] shard/test: Fix data race
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
96e690883f [#638] Unify test loggers
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
322c1dc273 [#638] Use test.NewLogger() in tests
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"

-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
82cc453be9 [#xx] shard: Fix data race in metrics tests
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 10:26:12 +00:00
4d9a6c07fb [#618] core: Replace fmt.Sprintf with strconv.FormatUint
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00