TrueCloudLab
/
frostfs-node
18
1
Fork 20
Code Issues 88 Pull Requests 7 Actions Releases Activity
4,359 Commits
7 Branches
20 Tags
47 MiB
Commit Graph

979 Commits (b118734909bd22402eb2192426d6470ff5520931)

Author SHA1 Message Date
Dmitrii Stepanov 8180a0664f [#887] node: Drop badger writecache implementation
Vulncheck / Vulncheck (pull_request) Successful in 6m1s Details
Build / Build Components (1.21) (pull_request) Successful in 7m37s Details
Build / Build Components (1.20) (pull_request) Successful in 7m52s Details
Tests and linters / Staticcheck (pull_request) Successful in 8m56s Details
Tests and linters / Lint (pull_request) Successful in 9m26s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 15m5s Details
Tests and linters / Tests with -race (pull_request) Successful in 15m7s Details
DCO action / DCO (pull_request) Successful in 1m1s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 4m1s Details 
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-22 13:00:54 +03:00
Anton Nikiforov d9cbb16bd3 [#866] Use TTL for blobovnicza tree cache 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-12-19 16:36:28 +00:00
Dmitrii Stepanov 7eb46404a1 [#863] blobovnicza: Fix counters
Vulncheck / Vulncheck (pull_request) Successful in 2m27s Details
DCO action / DCO (pull_request) Successful in 2m39s Details
Build / Build Components (1.21) (pull_request) Successful in 3m38s Details
Build / Build Components (1.20) (pull_request) Successful in 3m45s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m15s Details
Tests and linters / Lint (pull_request) Successful in 5m47s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 11m49s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 12m16s Details
Tests and linters / Tests with -race (pull_request) Successful in 12m31s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-13 13:34:29 +03:00
Dmitrii Stepanov 94ffe8bb45 [#857] golangci: Add testifylint linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-12 16:27:02 +03:00
Dmitrii Stepanov 3b7c0362a8 [#861] shard: Fix Delete object
DCO action / DCO (pull_request) Successful in 12m22s Details
Build / Build Components (1.20) (pull_request) Successful in 12m44s Details
Build / Build Components (1.21) (pull_request) Successful in 13m2s Details
Vulncheck / Vulncheck (pull_request) Successful in 13m3s Details
Tests and linters / Staticcheck (pull_request) Successful in 15m32s Details
Tests and linters / Lint (pull_request) Successful in 16m26s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 26m11s Details
Tests and linters / Tests with -race (pull_request) Successful in 28m29s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 3m28s Details 
It is possible that object doesn't exist in metabase.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-12 14:25:40 +03:00
Dmitrii Stepanov 681b2c5fd4 [#825] policer: Do not drop required linking objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-12 11:04:03 +00:00
Dmitrii Stepanov db49ad16cc [#826] blobovniczatree: Do not create DB's on init
DCO action / DCO (pull_request) Successful in 4m10s Details
Build / Build Components (1.21) (pull_request) Successful in 7m30s Details
Vulncheck / Vulncheck (pull_request) Successful in 7m5s Details
Tests and linters / Staticcheck (pull_request) Successful in 10m6s Details
Tests and linters / Lint (pull_request) Successful in 10m26s Details
Build / Build Components (1.20) (pull_request) Successful in 12m39s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 16m55s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 17m13s Details
Tests and linters / Tests with -race (pull_request) Successful in 17m14s Details 
Blobovniczas will be created on write requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov ad0697adc4 [#661] blobovnicza: Compute size with record size 
To get more accurate size of blobovnicza use record
size (lenght of key + lenght of data).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov e54dc3dc7c [#698] blobovnicza: Store counter values 
Blobovnicza initialization take a long time because of bucket
Stat() call. So now blobovnicza stores counters in META bucket.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov 5e8c08da3e [#661] blobstore: Add address to error logs 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov 8911656b1a [#661] metrcis: Add rebuild percent metric 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov 2407e5f5ff [#661] blobovniczatree: Do not sort DB's and indicies 
Put stores object to next active DB, so there is no need to sort DBs.
In addition, it adds unnecessary DB openings.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov c6a739e746 [#661] blobovniczatree: Make Rebuild concurrent for objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov f1c7905263 [#661] blobovniczatree: Make Rebuild concurrent 
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov d4d905ecc6 [#661] metrics: Add blobovniczatree rebuild metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov b2769ca3de [#661] blobovniczatree: Make Rebuild failover safe 
Now move info stores in blobovnicza, so in case of failover
rebuild completes previous operation first.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov da4fee2d0b [#698] blobovniczatree: Init blobovniczas concurrently 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:33 +03:00
Dmitrii Stepanov 422226da18 [#661] blobovniczatree: Add Rebuild implementation 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov a531eaf8bc [#661] blobstor: Add Rebuild implementation 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov c1667a11d2 [#661] blobovniczatree: Allow to change depth or width 
Now it is possible to change depth or with of blobovniczatree.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov 484eb59893 [#661] blobovniczatree: Use .db extension for db files 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov 44552a849b [#661] shard: Add blobstor rebuilder 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov a478050639 [#838] metabase: Resolve funlen linter
DCO action / DCO (pull_request) Successful in 1m59s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m0s Details
Build / Build Components (1.21) (pull_request) Successful in 3m40s Details
Tests and linters / Lint (pull_request) Successful in 4m57s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m38s Details
Build / Build Components (1.20) (pull_request) Successful in 5m17s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m1s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m39s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m46s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov d30ab5f29e [#838] metabase: Count user objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov f314da4af3 [#838] metabase: Add user object type counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov 29550fe600 [#838] shard: Refactor `updateMetrics` method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Alexander Chuprov b2c63e57ba [#651] engine/test: Speedup StorageEngine_Inhume 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-30 13:19:43 +00:00
Alexander Chuprov 445ebcc0e7 [#651] shard/test: Speedup Shard_Delete 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-30 13:19:43 +00:00
Alexander Chuprov 2302e5d342 [#651] shard/test: Refactor Shard_Delete 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-30 13:19:43 +00:00
Dmitrii Stepanov 07390ad4e3 [#715] node: Unify config parameter names
DCO action / DCO (pull_request) Successful in 2m54s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m11s Details
Tests and linters / Staticcheck (pull_request) Successful in 3m55s Details
Build / Build Components (1.21) (pull_request) Successful in 3m51s Details
Build / Build Components (1.20) (pull_request) Successful in 4m6s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 5m10s Details
Tests and linters / Lint (pull_request) Successful in 5m24s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m56s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m46s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-22 17:13:50 +03:00
Dmitrii Stepanov 8d18fa159e [#667] writecache: Fix flush test
DCO action / DCO (pull_request) Successful in 1m28s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m55s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m15s Details
Build / Build Components (1.21) (pull_request) Successful in 4m40s Details
Build / Build Components (1.20) (pull_request) Successful in 4m54s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m32s Details
Tests and linters / Lint (pull_request) Successful in 5m59s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m59s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m57s Details 
Allow to disable background flusher for testing purposes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-17 17:45:43 +03:00
Dmitrii Stepanov 76ff26039c [#96] node: Drop neo-go's `slices` package
DCO action / DCO (pull_request) Successful in 2m13s Details
Build / Build Components (1.20) (pull_request) Successful in 5m48s Details
Build / Build Components (1.21) (pull_request) Successful in 5m47s Details
Vulncheck / Vulncheck (pull_request) Successful in 5m47s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 7m33s Details
Tests and linters / Staticcheck (pull_request) Successful in 7m37s Details
Tests and linters / Lint (pull_request) Successful in 8m8s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 10m38s Details
Tests and linters / Tests with -race (pull_request) Successful in 11m22s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-17 13:24:04 +03:00
Airat Arifullin 47286ebf32 [#805] pilorama: Fix TreeDrop 
* If treeID is empty then deleting buckets for cursor may get
  invalidated. So, buckets should be gathered before deleting.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-17 10:21:35 +00:00
Dmitrii Stepanov 29fe8c41f3 [#655] storage: Drop `ErrorHandler`
DCO action / DCO (pull_request) Successful in 1m42s Details
Build / Build Components (1.21) (pull_request) Successful in 3m23s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m25s Details
Build / Build Components (1.20) (pull_request) Successful in 4m16s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m24s Details
Tests and linters / Lint (pull_request) Successful in 5m2s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m36s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m42s Details
Tests and linters / Tests with -race (pull_request) Successful in 7m18s Details 
The only one usage was for logging.
Now logging performed by storage anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-16 17:27:38 +03:00
Dmitrii Stepanov 137e987a4e [#655] storage: Drop `LazyHandler` 
LazyHandler is implemented and used incorrectly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-16 17:27:38 +03:00
Dmitrii Stepanov 1b22801eed [#788] engine: Fix flaky tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-09 10:25:46 +03:00
Dmitrii Stepanov 70ab1ebd54 [#763] metrics: Add container_objects_total metric
DCO action / DCO (pull_request) Successful in 3m54s Details
Build / Build Components (1.20) (pull_request) Successful in 4m58s Details
Build / Build Components (1.21) (pull_request) Successful in 5m16s Details
Vulncheck / Vulncheck (pull_request) Successful in 9m54s Details
Tests and linters / Lint (pull_request) Successful in 10m57s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 12m40s Details
Tests and linters / Staticcheck (pull_request) Successful in 12m34s Details
Tests and linters / Tests with -race (pull_request) Successful in 12m48s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 13m19s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov 9c98fa6152 [#763] metabase: Add container objects counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov 226e84d782 [#684] node: Add skipped objects count to evacuation result 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-07 12:17:11 +00:00
Dmitrii Stepanov c80b46fad3 [#754] blobstor: Estimate compressability
DCO action / DCO (pull_request) Successful in 1m59s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m31s Details
Build / Build Components (1.20) (pull_request) Successful in 4m37s Details
Build / Build Components (1.21) (pull_request) Successful in 4m33s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 4m54s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m49s Details
Tests and linters / Tests with -race (pull_request) Successful in 5m9s Details
Tests and linters / Lint (pull_request) Successful in 6m4s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m9s Details 
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-01 11:24:32 +03:00
Dmitrii Stepanov 79088baa06 [#772] node: Apply gofumpt 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:03 +03:00
Evgenii Stratonikov 00aa6d9749 [#633] shard/test: Fix TestCounters() 
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Dmitrii Stepanov 869518be0a [#728] writecache: Fix Badger writecache race.
Build / Build Components (1.21) (pull_request) Successful in 1m47s Details
DCO action / DCO (pull_request) Successful in 2m1s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m5s Details
Build / Build Components (1.20) (pull_request) Successful in 3m58s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m43s Details
Tests and linters / Staticcheck (pull_request) Successful in 5m49s Details
Tests and linters / Lint (pull_request) Successful in 6m20s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m8s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 3m6s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-30 18:36:41 +03:00
Dmitrii Stepanov d4b6ebe7e7 [#725] writecache: Fix metric values
DCO action / DCO (pull_request) Successful in 1m33s Details
Build / Build Components (1.21) (pull_request) Successful in 3m11s Details
Build / Build Components (1.20) (pull_request) Successful in 3m22s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m34s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m0s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m1s Details
Tests and linters / Staticcheck (pull_request) Successful in 6m4s Details
Tests and linters / Tests with -race (pull_request) Successful in 3m58s Details
Tests and linters / Lint (pull_request) Successful in 11m43s Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-27 12:22:29 +03:00
Anton Nikiforov 559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Anton Nikiforov c0b86f2d93 [#642] writecache: Remove usage of close channel in badger 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Dmitrii Stepanov f2437f7ae9 [#734] shard: Fix Delete method
DCO action / DCO (pull_request) Successful in 1m36s Details
Build / Build Components (1.20) (pull_request) Successful in 3m46s Details
Vulncheck / Vulncheck (pull_request) Successful in 3m16s Details
Tests and linters / Staticcheck (pull_request) Successful in 4m42s Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m35s Details
Tests and linters / Tests with -race (pull_request) Successful in 6m33s Details
Build / Build Components (1.21) (pull_request) Successful in 13m7s Details
Tests and linters / Lint (pull_request) Successful in 19m14s Details 
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov 7e0c5a55de [#734] writecache: Fix flush 
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:17 +03:00