Commit Graph

1054 Commits (1eb47ab2cec34b7674bbdcfc6b1b32b2e1f0cdf8)

Author SHA1 Message Date
Dmitrii Stepanov cfc5ce7853 [#964] metabase: Drop GC marks if object not found
GC inhumes expired locks and tombstones on all the shards.
So it could be GC mark without object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-08 07:54:39 +00:00
Dmitrii Stepanov 9ba48c582d [#917] engine: Allow to detach shards
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-06 14:49:47 +03:00
Anton Nikiforov d0eadf7ea2 [#799] engine: Skip put when object removed from shard
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-01 17:49:22 +00:00
Anton Nikiforov e3573de6db [#930] gc: Stop internal activity by context
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
Dmitrii Stepanov 675eec91f3 [#938] shard: Update only changed counters
DCO action / DCO (pull_request) Successful in 5m40s Details
Vulncheck / Vulncheck (pull_request) Successful in 5m33s Details
Tests and linters / Staticcheck (pull_request) Successful in 7m12s Details
Build / Build Components (1.21) (pull_request) Successful in 7m25s Details
Build / Build Components (1.20) (pull_request) Successful in 7m29s Details
Tests and linters / Lint (pull_request) Successful in 8m26s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m48s Details
Tests and linters / Tests with -race (pull_request) Successful in 9m56s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 10m10s Details
If metric value hasn't changed, but we update metric, then
non existed metric will apear with zero value.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
Dmitrii Stepanov c681354afd [#938] engine: Fix container count removal
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
Dmitrii Stepanov 5ed330e436 [#927] metabase: Delete GC marks
DCO action / DCO (pull_request) Successful in 6m34s Details
Vulncheck / Vulncheck (pull_request) Successful in 7m19s Details
Build / Build Components (1.20) (pull_request) Successful in 7m48s Details
Build / Build Components (1.21) (pull_request) Successful in 9m39s Details
Tests and linters / Staticcheck (pull_request) Successful in 10m36s Details
Tests and linters / Lint (pull_request) Successful in 11m26s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 12m57s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 13m8s Details
Tests and linters / Tests with -race (pull_request) Successful in 13m7s Details
`key` is changed inside `db.get`, so encode address again after get.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-24 18:51:16 +03:00
Dmitrii Stepanov 931a5e9aaf [#918] engine: Move shard to degraded mode if metabase open failed
DCO action / DCO (pull_request) Successful in 1m55s Details
Build / Build Components (1.20) (pull_request) Successful in 2m14s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m15s Details
Build / Build Components (1.21) (pull_request) Successful in 4m6s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m58s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m24s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m11s Details
Tests and linters / Lint (pull_request) Successful in 6m37s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m35s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 11:16:40 +03:00
Dmitrii Stepanov f526f49995 [#874] engine: Check object existance concurrently
DCO action / DCO (pull_request) Successful in 1m36s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m33s Details
Build / Build Components (1.21) (pull_request) Successful in 3m9s Details
Build / Build Components (1.20) (pull_request) Successful in 4m49s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m5s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m6s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m39s Details
Tests and linters / Lint (pull_request) Successful in 7m0s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m46s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 09:28:29 +03:00
Dmitrii Stepanov f5160b27fc [#920] tests: Fix data races
DCO action / DCO (pull_request) Successful in 2m2s Details
Build / Build Components (1.21) (pull_request) Successful in 3m35s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m3s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m23s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m33s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m41s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m47s Details
Tests and linters / Lint (pull_request) Successful in 8m12s Details
Build / Build Components (1.20) (pull_request) Successful in 1m32s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-19 14:06:05 +03:00
Dmitrii Stepanov 63d3ed1ad8 [#904] tests: Close test engine after test
DCO action / DCO (pull_request) Successful in 2m7s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m37s Details
Build / Build Components (1.20) (pull_request) Successful in 4m0s Details
Build / Build Components (1.21) (pull_request) Successful in 3m56s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m25s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m22s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m43s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m47s Details
Tests and linters / Lint (pull_request) Successful in 7m5s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
Dmitrii Stepanov 57171907e3 [#904] metabase: Return if object was actuall inserted
This requires to count metrics properly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
Dmitrii Stepanov c1a80235db [#904] metabase: Log Inhume operation
It will be very useful for troubleshooting.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 18:42:52 +03:00
Dmitrii Stepanov a2ab373a0a [#895] metabase: Do not delete GC mark for virtual objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
Dmitrii Stepanov 7166e77c2b [#895] test: Add logger to test shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
Dmitrii Stepanov 47dcfa20f3 [#895] test: Use t.Cleanup only for external resources
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
Dmitrii Stepanov f1b2b8bffa [#895] test: Fix NewLogger arguments list
`debug` is always true.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
Dmitrii Stepanov 4b8b4da681 [#864] engine: Drop container count metric if container removed
DCO action / DCO (pull_request) Successful in 1m30s Details
Build / Build Components (1.21) (pull_request) Successful in 3m29s Details
Build / Build Components (1.20) (pull_request) Successful in 3m53s Details
Tests and linters / Lint (pull_request) Successful in 4m31s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m1s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m51s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m13s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m34s Details
Vulncheck / Vulncheck (pull_request) Successful in 58s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:45:32 +03:00
Dmitrii Stepanov d75e7e9a21 [#864] engine: Drop container size metric if container deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:44:54 +03:00
Dmitrii Stepanov dfd62ca6b1 [#864] metabase: Refactor delete/inhume
Available -> Logic, Raw -> Phy for delete/inhume results.
Use single counter instead of vectors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-09 09:59:42 +03:00
Evgenii Stratonikov 225fe2d4d5 [#894] blobovniczatree: Speedup rebuild test
Down from 3s to 300ms.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-12-29 16:28:54 +00:00
Dmitrii Stepanov 581887148a [#569] cli: Add `control shards writecache seal` command
DCO action / DCO (pull_request) Successful in 2m46s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m3s Details
Build / Build Components (1.21) (pull_request) Successful in 3m53s Details
Build / Build Components (1.20) (pull_request) Successful in 4m0s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m31s Details
Tests and linters / Lint (pull_request) Successful in 6m17s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 12m47s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 13m14s Details
Tests and linters / Tests with -race (pull_request) Successful in 13m12s Details
It does the same as `control shards flush-writecache --seal`, but
has better name.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
Dmitrii Stepanov 7a9db5bcdd [#569] writecache: Do not wait modeMtx if mode changes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
Dmitrii Stepanov 32c282ca10 [#569] writecache: Refactor flush
Make single RUnlock call instead of two.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
Dmitrii Stepanov 0cb0fc1735 [#569] writecache: Allow to seal writecache after flush
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
Dmitrii Stepanov 8180a0664f [#887] node: Drop badger writecache implementation
Vulncheck / Vulncheck (pull_request) Successful in 6m1s Details
Build / Build Components (1.21) (pull_request) Successful in 7m37s Details
Build / Build Components (1.20) (pull_request) Successful in 7m52s Details
Tests and linters / Staticcheck (pull_request) Successful in 8m56s Details
Tests and linters / Lint (pull_request) Successful in 9m26s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 15m5s Details
Tests and linters / Tests with -race (pull_request) Successful in 15m7s Details
DCO action / DCO (pull_request) Successful in 1m1s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 4m1s Details
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-22 13:00:54 +03:00
Anton Nikiforov d9cbb16bd3 [#866] Use TTL for blobovnicza tree cache
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-12-19 16:36:28 +00:00
Dmitrii Stepanov 7eb46404a1 [#863] blobovnicza: Fix counters
Vulncheck / Vulncheck (pull_request) Successful in 2m27s Details
DCO action / DCO (pull_request) Successful in 2m39s Details
Build / Build Components (1.21) (pull_request) Successful in 3m38s Details
Build / Build Components (1.20) (pull_request) Successful in 3m45s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m15s Details
Tests and linters / Lint (pull_request) Successful in 5m47s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 11m49s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 12m16s Details
Tests and linters / Tests with -race (pull_request) Successful in 12m31s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-13 13:34:29 +03:00
Dmitrii Stepanov 94ffe8bb45 [#857] golangci: Add testifylint linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
Dmitrii Stepanov 3b7c0362a8 [#861] shard: Fix Delete object
DCO action / DCO (pull_request) Successful in 12m22s Details
Build / Build Components (1.20) (pull_request) Successful in 12m44s Details
Build / Build Components (1.21) (pull_request) Successful in 13m2s Details
Vulncheck / Vulncheck (pull_request) Successful in 13m3s Details
Tests and linters / Staticcheck (pull_request) Successful in 15m32s Details
Tests and linters / Lint (pull_request) Successful in 16m26s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 26m11s Details
Tests and linters / Tests with -race (pull_request) Successful in 28m29s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 3m28s Details
It is possible that object doesn't exist in metabase.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 14:25:40 +03:00
Dmitrii Stepanov 681b2c5fd4 [#825] policer: Do not drop required linking objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 11:04:03 +00:00
Dmitrii Stepanov db49ad16cc [#826] blobovniczatree: Do not create DB's on init
DCO action / DCO (pull_request) Successful in 4m10s Details
Build / Build Components (1.21) (pull_request) Successful in 7m30s Details
Vulncheck / Vulncheck (pull_request) Successful in 7m5s Details
Tests and linters / Staticcheck (pull_request) Successful in 10m6s Details
Tests and linters / Lint (pull_request) Successful in 10m26s Details
Build / Build Components (1.20) (pull_request) Successful in 12m39s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 16m55s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 17m13s Details
Tests and linters / Tests with -race (pull_request) Successful in 17m14s Details
Blobovniczas will be created on write requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov ad0697adc4 [#661] blobovnicza: Compute size with record size
To get more accurate size of blobovnicza use record
size (lenght of key + lenght of data).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov e54dc3dc7c [#698] blobovnicza: Store counter values
Blobovnicza initialization take a long time because of bucket
Stat() call. So now blobovnicza stores counters in META bucket.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov 5e8c08da3e [#661] blobstore: Add address to error logs
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov 8911656b1a [#661] metrcis: Add rebuild percent metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov 2407e5f5ff [#661] blobovniczatree: Do not sort DB's and indicies
Put stores object to next active DB, so there is no need to sort DBs.
In addition, it adds unnecessary DB openings.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov c6a739e746 [#661] blobovniczatree: Make Rebuild concurrent for objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov f1c7905263 [#661] blobovniczatree: Make Rebuild concurrent
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov d4d905ecc6 [#661] metrics: Add blobovniczatree rebuild metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov b2769ca3de [#661] blobovniczatree: Make Rebuild failover safe
Now move info stores in blobovnicza, so in case of failover
rebuild completes previous operation first.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov da4fee2d0b [#698] blobovniczatree: Init blobovniczas concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
Dmitrii Stepanov 422226da18 [#661] blobovniczatree: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
Dmitrii Stepanov a531eaf8bc [#661] blobstor: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
Dmitrii Stepanov c1667a11d2 [#661] blobovniczatree: Allow to change depth or width
Now it is possible to change depth or with of blobovniczatree.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
Dmitrii Stepanov 484eb59893 [#661] blobovniczatree: Use .db extension for db files
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
Dmitrii Stepanov 44552a849b [#661] shard: Add blobstor rebuilder
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
Dmitrii Stepanov a478050639 [#838] metabase: Resolve funlen linter
DCO action / DCO (pull_request) Successful in 1m59s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m0s Details
Build / Build Components (1.21) (pull_request) Successful in 3m40s Details
Tests and linters / Lint (pull_request) Successful in 4m57s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m38s Details
Build / Build Components (1.20) (pull_request) Successful in 5m17s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m1s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m39s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m46s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
Dmitrii Stepanov d30ab5f29e [#838] metabase: Count user objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
Dmitrii Stepanov f314da4af3 [#838] metabase: Add user object type counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
Dmitrii Stepanov 29550fe600 [#838] shard: Refactor `updateMetrics` method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
Alexander Chuprov b2c63e57ba [#651] engine/test: Speedup StorageEngine_Inhume
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
Alexander Chuprov 445ebcc0e7 [#651] shard/test: Speedup Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
Alexander Chuprov 2302e5d342 [#651] shard/test: Refactor Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
Dmitrii Stepanov 07390ad4e3 [#715] node: Unify config parameter names
DCO action / DCO (pull_request) Successful in 2m54s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m11s Details
Tests and linters / Staticcheck (pull_request) Successful in 3m55s Details
Build / Build Components (1.21) (pull_request) Successful in 3m51s Details
Build / Build Components (1.20) (pull_request) Successful in 4m6s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m10s Details
Tests and linters / Lint (pull_request) Successful in 5m24s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m56s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m46s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 17:13:50 +03:00
Dmitrii Stepanov 8d18fa159e [#667] writecache: Fix flush test
DCO action / DCO (pull_request) Successful in 1m28s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m55s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m15s Details
Build / Build Components (1.21) (pull_request) Successful in 4m40s Details
Build / Build Components (1.20) (pull_request) Successful in 4m54s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m32s Details
Tests and linters / Lint (pull_request) Successful in 5m59s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m59s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m57s Details
Allow to disable background flusher for testing purposes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 17:45:43 +03:00
Dmitrii Stepanov 76ff26039c [#96] node: Drop neo-go's `slices` package
DCO action / DCO (pull_request) Successful in 2m13s Details
Build / Build Components (1.20) (pull_request) Successful in 5m48s Details
Build / Build Components (1.21) (pull_request) Successful in 5m47s Details
Vulncheck / Vulncheck (pull_request) Successful in 5m47s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m33s Details
Tests and linters / Staticcheck (pull_request) Successful in 7m37s Details
Tests and linters / Lint (pull_request) Successful in 8m8s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 10m38s Details
Tests and linters / Tests with -race (pull_request) Successful in 11m22s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 13:24:04 +03:00
Airat Arifullin 47286ebf32 [#805] pilorama: Fix TreeDrop
* If treeID is empty then deleting buckets for cursor may get
  invalidated. So, buckets should be gathered before deleting.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-17 10:21:35 +00:00
Dmitrii Stepanov 29fe8c41f3 [#655] storage: Drop `ErrorHandler`
DCO action / DCO (pull_request) Successful in 1m42s Details
Build / Build Components (1.21) (pull_request) Successful in 3m23s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m25s Details
Build / Build Components (1.20) (pull_request) Successful in 4m16s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m24s Details
Tests and linters / Lint (pull_request) Successful in 5m2s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m36s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m42s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m18s Details
The only one usage was for logging.
Now logging performed by storage anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
Dmitrii Stepanov 137e987a4e [#655] storage: Drop `LazyHandler`
LazyHandler is implemented and used incorrectly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
Dmitrii Stepanov 1b22801eed [#788] engine: Fix flaky tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:25:46 +03:00
Dmitrii Stepanov 70ab1ebd54 [#763] metrics: Add container_objects_total metric
DCO action / DCO (pull_request) Successful in 3m54s Details
Build / Build Components (1.20) (pull_request) Successful in 4m58s Details
Build / Build Components (1.21) (pull_request) Successful in 5m16s Details
Vulncheck / Vulncheck (pull_request) Successful in 9m54s Details
Tests and linters / Lint (pull_request) Successful in 10m57s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 12m40s Details
Tests and linters / Staticcheck (pull_request) Successful in 12m34s Details
Tests and linters / Tests with -race (pull_request) Successful in 12m48s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 13m19s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
Dmitrii Stepanov 9c98fa6152 [#763] metabase: Add container objects counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
Dmitrii Stepanov 226e84d782 [#684] node: Add skipped objects count to evacuation result
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
Dmitrii Stepanov c80b46fad3 [#754] blobstor: Estimate compressability
DCO action / DCO (pull_request) Successful in 1m59s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m31s Details
Build / Build Components (1.20) (pull_request) Successful in 4m37s Details
Build / Build Components (1.21) (pull_request) Successful in 4m33s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 4m54s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m49s Details
Tests and linters / Tests with -race (pull_request) Successful in 5m9s Details
Tests and linters / Lint (pull_request) Successful in 6m4s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m9s Details
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 11:24:32 +03:00
Dmitrii Stepanov 79088baa06 [#772] node: Apply gofumpt
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
Evgenii Stratonikov 00aa6d9749 [#633] shard/test: Fix TestCounters()
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
Evgenii Stratonikov b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
Dmitrii Stepanov 869518be0a [#728] writecache: Fix Badger writecache race.
Build / Build Components (1.21) (pull_request) Successful in 1m47s Details
DCO action / DCO (pull_request) Successful in 2m1s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m5s Details
Build / Build Components (1.20) (pull_request) Successful in 3m58s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m43s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m49s Details
Tests and linters / Lint (pull_request) Successful in 6m20s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m8s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 3m6s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-30 18:36:41 +03:00
Dmitrii Stepanov d4b6ebe7e7 [#725] writecache: Fix metric values
DCO action / DCO (pull_request) Successful in 1m33s Details
Build / Build Components (1.21) (pull_request) Successful in 3m11s Details
Build / Build Components (1.20) (pull_request) Successful in 3m22s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m34s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m0s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m1s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m4s Details
Tests and linters / Tests with -race (pull_request) Successful in 3m58s Details
Tests and linters / Lint (pull_request) Successful in 11m43s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-27 12:22:29 +03:00
Anton Nikiforov 559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:57:50 +00:00
Anton Nikiforov c0b86f2d93 [#642] writecache: Remove usage of close channel in badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:57:50 +00:00
Dmitrii Stepanov f2437f7ae9 [#734] shard: Fix Delete method
DCO action / DCO (pull_request) Successful in 1m36s Details
Build / Build Components (1.20) (pull_request) Successful in 3m46s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m16s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m42s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m35s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m33s Details
Build / Build Components (1.21) (pull_request) Successful in 13m7s Details
Tests and linters / Lint (pull_request) Successful in 19m14s Details
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
Dmitrii Stepanov f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
Dmitrii Stepanov 7e0c5a55de [#734] writecache: Fix flush
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:17 +03:00
Dmitrii Stepanov d07afd803c [#726] writecache: Fix small object flush for Badger
DCO action / DCO (pull_request) Successful in 2m25s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m41s Details
Build / Build Components (1.21) (pull_request) Successful in 3m2s Details
Build / Build Components (1.20) (pull_request) Successful in 3m17s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m26s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m9s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 18m15s Details
Tests and linters / Lint (pull_request) Successful in 20m21s Details
Tests and linters / Tests with -race (pull_request) Successful in 18m2s Details
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:50 +03:00
Dmitrii Stepanov 997ac7cd8d [#726] writecache: Fix small object flush for BBolt
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:44 +03:00
Alexander Chuprov c1e4130020 [#146] node: Add trace_id to logs
Vulncheck / Vulncheck (pull_request) Successful in 3m7s Details
DCO action / DCO (pull_request) Successful in 3m36s Details
Build / Build Components (1.21) (pull_request) Successful in 3m29s Details
Build / Build Components (1.20) (pull_request) Successful in 3m37s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m39s Details
Tests and linters / Lint (pull_request) Successful in 5m2s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m29s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s Details
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
Airat Arifullin 3889e829e6 [#667] writecache: Add logs for report error func in tests
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-09-14 17:02:54 +00:00
Evgenii Stratonikov c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch
DCO action / DCO (pull_request) Successful in 1m43s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m0s Details
Tests and linters / Staticcheck (pull_request) Successful in 2m54s Details
Build / Build Components (1.20) (pull_request) Successful in 2m49s Details
Build / Build Components (1.21) (pull_request) Successful in 2m44s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 3m43s Details
Tests and linters / Lint (pull_request) Successful in 3m58s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 23m28s Details
Tests and linters / Tests with -race (pull_request) Successful in 29m0s Details
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-14 10:39:18 +03:00
Ekaterina Lebedeva 8a81af5a3b [#653] Add context parameter to Open functions
DCO action / DCO (pull_request) Successful in 1m38s Details
Build / Build Components (1.20) (pull_request) Successful in 4m22s Details
Build / Build Components (1.21) (pull_request) Successful in 4m25s Details
Vulncheck / Vulncheck (pull_request) Successful in 4m56s Details
Tests and linters / Lint (pull_request) Successful in 6m1s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s Details
Tests and linters / Staticcheck (pull_request) Successful in 8m1s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s Details
Tests and linters / Tests with -race (pull_request) Successful in 8m32s Details
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
Evgenii Stratonikov a716db99db [#668] shard/test: Do not alter rootPath option
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov 36759f8434 [#668] shard/test: Properly check event processing
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov 39879fa868 [#668] shard/test: Add dontRelease options
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov c661ba1312 [#668] shard/test: Use sane defaults in the test constructor
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov 268adb79cb [#668] shard/test: Simplify shard construction
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov 429f941cda [#668] shard/test: Release shard in t.Cleanup()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov 382eb8a485 [#668] shard/test: Disable GC where it is not needed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov 42696016de [#668] shard: Close stopChannel in GC
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov bdecfbc1be [#668] shard/test: Move tests to the main package
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov aa23c6a83a [#668] shard/test: Remove subtest from TestCounters
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov da8f384324 [#668] shard/test: Fix typo in `existence`
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Dmitrii Stepanov 7456c8556a [#536] blobovnicza: Add blobovniczatree DB cache
DCO action / DCO (pull_request) Successful in 1m33s Details
Vulncheck / Vulncheck (pull_request) Successful in 2m43s Details
Build / Build Components (1.20) (pull_request) Successful in 3m40s Details
Tests and linters / Staticcheck (pull_request) Successful in 3m59s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m44s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m3s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m28s Details
Build / Build Components (1.21) (pull_request) Successful in 12m34s Details
Tests and linters / Lint (pull_request) Successful in 12m51s Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:53:11 +03:00
Dmitrii Stepanov c672f59ab8 [#536] blobovnicza: Drop cache
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:51:26 +03:00
Evgenii Stratonikov b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-31 08:39:42 +00:00
Anton Nikiforov 4dff9555f1 [#568] writecache: Improve flushing scheme for badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:22:28 +00:00
Alejandro Lopez 1daef2ceeb [#660] writecache: Fix remaining addr2key uses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-30 17:12:33 +00:00
Evgenii Stratonikov 9072772a09 [#649] shard/test: Increase GC remover interval
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
Evgenii Stratonikov c4db8e7690 [#637] shard/test: Fix data race
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
Evgenii Stratonikov 96e690883f [#638] Unify test loggers
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00