acid-ant/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
4945 commits 110 branches 26 tags 20 MiB
Commit graph

1161 commits

Author SHA1 Message Date
Dmitrii Stepanov
a531eaf8bc [#661] blobstor: Add Rebuild implementation 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov
c1667a11d2 [#661] blobovniczatree: Allow to change depth or width 
Now it is possible to change depth or with of blobovniczatree.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov
484eb59893 [#661] blobovniczatree: Use .db extension for db files 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov
44552a849b [#661] shard: Add blobstor rebuilder 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-07 15:37:32 +03:00
Dmitrii Stepanov
a478050639 [#838] metabase: Resolve funlen linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov
d30ab5f29e [#838] metabase: Count user objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov
f314da4af3 [#838] metabase: Add user object type counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Dmitrii Stepanov
29550fe600 [#838] shard: Refactor updateMetrics method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-06 15:44:21 +03:00
Alexander Chuprov
b2c63e57ba [#651] engine/test: Speedup StorageEngine_Inhume 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-30 13:19:43 +00:00
Alexander Chuprov
445ebcc0e7 [#651] shard/test: Speedup Shard_Delete 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-30 13:19:43 +00:00
Alexander Chuprov
2302e5d342 [#651] shard/test: Refactor Shard_Delete 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-11-30 13:19:43 +00:00
Dmitrii Stepanov
07390ad4e3 [#715] node: Unify config parameter names 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-22 17:13:50 +03:00
Dmitrii Stepanov
8d18fa159e [#667] writecache: Fix flush test 
Allow to disable background flusher for testing purposes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-17 17:45:43 +03:00
Dmitrii Stepanov
76ff26039c [#96] node: Drop neo-go's slices package 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-17 13:24:04 +03:00
aarifullin
47286ebf32 [#805] pilorama: Fix TreeDrop 
* If treeID is empty then deleting buckets for cursor may get
  invalidated. So, buckets should be gathered before deleting.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-17 10:21:35 +00:00
Dmitrii Stepanov
29fe8c41f3 [#655] storage: Drop ErrorHandler 
The only one usage was for logging.
Now logging performed by storage anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-16 17:27:38 +03:00
Dmitrii Stepanov
137e987a4e [#655] storage: Drop LazyHandler 
LazyHandler is implemented and used incorrectly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-16 17:27:38 +03:00
Dmitrii Stepanov
1b22801eed [#788] engine: Fix flaky tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-09 10:25:46 +03:00
Dmitrii Stepanov
70ab1ebd54 [#763] metrics: Add container_objects_total metric 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov
9c98fa6152 [#763] metabase: Add container objects counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov
226e84d782 [#684] node: Add skipped objects count to evacuation result 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-07 12:17:11 +00:00
Dmitrii Stepanov
c80b46fad3 [#754] blobstor: Estimate compressability 
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-01 11:24:32 +03:00
Dmitrii Stepanov
79088baa06 [#772] node: Apply gofumpt 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:03 +03:00
Evgenii Stratonikov
00aa6d9749 [#633] shard/test: Fix TestCounters() 
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Dmitrii Stepanov
869518be0a [#728] writecache: Fix Badger writecache race. 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-30 18:36:41 +03:00
Dmitrii Stepanov
d4b6ebe7e7 [#725] writecache: Fix metric values 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-27 12:22:29 +03:00
Anton Nikiforov
559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Anton Nikiforov
c0b86f2d93 [#642] writecache: Remove usage of close channel in badger 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Dmitrii Stepanov
f2437f7ae9 [#734] shard: Fix Delete method 
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov
7e0c5a55de [#734] writecache: Fix flush 
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:17 +03:00
Dmitrii Stepanov
d07afd803c [#726] writecache: Fix small object flush for Badger 
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-06 11:32:50 +03:00
Dmitrii Stepanov
997ac7cd8d [#726] writecache: Fix small object flush for BBolt 
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-06 11:32:44 +03:00
Alexander Chuprov
c1e4130020 [#146] node: Add trace_id to logs 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-09-27 11:05:27 +03:00
a.arifullin
3889e829e6 [#667] writecache: Add logs for report error func in tests 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-09-14 17:02:54 +00:00
Evgenii Stratonikov
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch 
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-14 10:39:18 +03:00
Ekaterina Lebedeva
8a81af5a3b [#653] Add context parameter to Open functions 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-09-07 18:03:29 +03:00
Evgenii Stratonikov
a716db99db [#668] shard/test: Do not alter rootPath option 
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
36759f8434 [#668] shard/test: Properly check event processing 
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
39879fa868 [#668] shard/test: Add dontRelease options 
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
268adb79cb [#668] shard/test: Simplify shard construction 
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
429f941cda [#668] shard/test: Release shard in t.Cleanup() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
382eb8a485 [#668] shard/test: Disable GC where it is not needed 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
42696016de [#668] shard: Close stopChannel in GC 
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
bdecfbc1be [#668] shard/test: Move tests to the main package 
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters 
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
da8f384324 [#668] shard/test: Fix typo in existence 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Dmitrii Stepanov
7456c8556a [#536] blobovnicza: Add blobovniczatree DB cache 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-09-01 13:53:11 +03:00
Dmitrii Stepanov
c672f59ab8 [#536] blobovnicza: Drop cache 
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-09-01 13:51:26 +03:00
Evgenii Stratonikov
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-31 08:39:42 +00:00
Anton Nikiforov
4dff9555f1 [#568] writecache: Improve flushing scheme for badger 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-08-30 17:22:28 +00:00
Alejandro Lopez
1daef2ceeb [#660] writecache: Fix remaining addr2key uses 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-30 17:12:33 +00:00
Evgenii Stratonikov
9072772a09 [#649] shard/test: Increase GC remover interval 
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:10:25 +00:00
Evgenii Stratonikov
c4db8e7690 [#637] shard/test: Fix data race 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:10:25 +00:00
Evgenii Stratonikov
96e690883f [#638] Unify test loggers 
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
322c1dc273 [#638] Use test.NewLogger() in tests 
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"

-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
82cc453be9 [#xx] shard: Fix data race in metrics tests 
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 10:26:12 +00:00
Alejandro Lopez
345a1a69a2 [#635] Use internal key type when deleting from badger wc 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-22 10:53:19 +03:00
Dmitrii Stepanov
354a92ea2c [#602] blobovnicza: Add leaf width implementation 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-21 10:27:32 +03:00
Alexander Chuprov
4d9a6c07fb [#618] core: Replace fmt.Sprintf with strconv.FormatUint 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-08-21 07:14:50 +00:00
Dmitrii Stepanov
f2811f8585 [#602] metrics: Add blobovnicza items counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-18 13:01:27 +03:00
Dmitrii Stepanov
c4e1d8eb07 [#602] node: Fix blobovnicza typos 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-18 11:14:10 +03:00
Dmitrii Stepanov
10e63537b2 [#602] metrics: Rename blobovnicza size metric 
`Size` is not size, but open db size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Dmitrii Stepanov
809e97626b [#602] blobovnicza: Fix size counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Dmitrii Stepanov
2e49d7ea7e [#602] blobovnicza: Init before using 
Fix blobovnicza size: after restart size metric resets.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Evgenii Stratonikov
f7042c5a6f [#609] Replace zaptest.NewLogger() with zap.L() 
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"

-zaptest.NewLogger(t)
+zap.L()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:51:10 +00:00
Evgenii Stratonikov
e604a3d749 [#607] *: Use zap.Stringer() where possible 
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
 f(
    ...,
-    zap.String(t, a.String()),
+    zap.Stringer(t, a),
    ...,
)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:50:02 +00:00
Evgenii Stratonikov
a8de37c8a2 [#607] *: Remove redundant if on error returns 
Semantic patch:
```
@@
@@
-if err != nil { return err }
-return nil
+return err
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:50:02 +00:00
Evgenii Stratonikov
5335e7089e [#615] pilorama: Speedup TestForest_ApplyRandom() 
Some of our pilorama tests fail on CI.
The reasons are not obvious, but one possible improvement
is using `WithNoSync` option for these. It should have much effect,
because we are writing on the tmpfs, but doesn't hurt anyway.

If I replace `t.TempDir()` with a local directory, test execution time
goes down from 5s (sync) to 0.4s (nosync), which is the same time as
with `t.TempDir()`. Maybe we have some strange CI configuration.

```
panic: test timed out after 10m0s
running tests:
	TestForest_ApplyRandom (8m22s)
	TestForest_ApplyRandom/bbolt (8m21s)
...
goroutine 170 [syscall]:
syscall.Syscall(0xc000100000?, 0xc00047b758?, 0x6aff9a?, 0xc00041c1b0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:69 +0x27
syscall.Fdatasync(0x9e35c0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/zsyscall_linux_amd64.go:418 +0x2a
go.etcd.io/bbolt.fdatasync(0xc000189000?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-16 19:09:08 +00:00
Dmitrii Stepanov
2efe9cc1be [#585] writecache: Fix DB counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
58c8722c81 [#585] fstree: Add optional file counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
baad49990c [#585] fstree: Return logical error if object deleted 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
0c52186572 [#585] fstree: Remove unused method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
eec97d177e [#585] writecache: Count items periodically 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Evgenii Stratonikov
d15199c5d8 [#596] engine: Consider context errors as logical 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-16 10:39:41 +03:00
Alejandro Lopez
88b6755c5e [#598] Fix use-after-close bug in badger writecache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-15 07:04:06 +00:00
Alejandro Lopez
ae8be495c8 [#xx] Avoid manual management of files in tests 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-14 14:01:39 +03:00
Alejandro Lopez
376f03a445 [#598] Hold mode mutex when setting mode 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-14 07:12:27 +00:00
Alexander Chuprov
21800e9fcc [#162] core: Move literals to constants 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-08-11 15:48:42 +03:00
Evgenii Stratonikov
b44a8dd46c [#597] *: Fix linter warnings 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-11 09:55:51 +03:00
Alejandro Lopez
8f994163ee [#586] Fix writecache benchmarks and refactor hacky NeedsCompression 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-10 08:05:18 +00:00
Alejandro Lopez
023b90342c [#584] Disable compression in badger writecache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-10 10:14:06 +03:00
Evgenii Stratonikov
d641cba2fc [#587] Do not use math/rand.Read 
Fix staticcheck warnings after go1.20 update.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-09 16:02:44 +03:00
Alejandro Lopez
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 10:26:53 +00:00
Alejandro Lopez
ae322e9f73 [#576] Set SyncWrites for badger writecache by default 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 11:28:26 +03:00
Alejandro Lopez
7da4306e38 [#575] writecache: Fix log level for badger writecache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-08 08:16:36 +00:00
Dmitrii Stepanov
0e697266c3 [#563] writecache: Fix metrics and bolt delete 
Estimate cache size after delete objects to update metric.
Update counters on small object deletion.
Do not count bbolt DB file as FSTree object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-07 12:53:28 +00:00
Alejandro Lopez
1a0cb0f34a [#421] Try using badger for the write-cache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-07 08:16:57 +00:00
Alejandro Lopez
65c72f3e0b [#559] Remove manual path handling in fstree tests 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-03 10:03:41 +03:00
Evgenii Stratonikov
35dc64bd7b [#547] metabase: Fix datarace in tests 
Quite an old one bf9e938a3b.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-31 15:35:15 +00:00
Evgenii Stratonikov
ec8b4fdc48 [#541] writecache/test: Close writecache on exit 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-29 10:57:25 +00:00
Evgenii Stratonikov
ad5f527bd3 [#541] writecache/test: Remove initWC() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-29 10:57:25 +00:00
Dmitrii Stepanov
af82c2865e [#335] treesvc: Fix inmemory unit tests and nil meta items 
Bolt forest saves empty slice of items. Now inmemory forest
does it the same way.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-07-20 10:14:10 +03:00
Dmitrii Stepanov
b4e72a2dfd [#335] treesvc: Sort nodes by Filename in GetSubTree 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-07-20 10:14:10 +03:00
Dmitrii Stepanov
3e8de14e7d [#382] evacuate: Fix unit tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-07-17 10:24:05 +00:00
Alejandro Lopez
11027945d8 [#479] writecache: Fix writecache fstree flush premature ctx cancel 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-14 10:25:52 +03:00
Alexander Chuprov
033eaf77e1 [#496] node: Fix linter importas 
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-07-06 15:36:41 +03:00
Evgenii Stratonikov
cab51c8cbe [#1] metabase: Rename blindlyProcess() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-28 12:13:20 +00:00
Evgenii Stratonikov
f4c71cea65 [#1] *: Replace outdated FIXME/TODO links 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-28 12:13:20 +00:00
Dmitrii Stepanov
43d263c3d5 [#428] linter: Fix unkeyed assignment 
Thanks to gopls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-26 13:29:39 +00:00
Dmitrii Stepanov
cac4ed93d6 [#428] engine: Add low_mem config parameter 
Concurrent initialization in case of the metabase resync leads to
high memory consumption and potential OOM.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-26 13:29:39 +00:00
Evgenii Stratonikov
4bf345225c [#447] pilorama: Use named constant for the key size 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-26 07:42:05 +00:00
Evgenii Stratonikov
b4ce0b0412 [#447] pilorama: Do not undo log for create ops 
```
goos: linux
goarch: amd64
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
         │     old      │                 new                  │
         │    sec/op    │    sec/op     vs base                │
Create-8   36.48µ ± 11%   30.34µ ± 14%  -16.84% (p=0.000 n=10)

         │     old      │                 new                  │
         │     B/op     │     B/op      vs base                │
Create-8   43.01Ki ± 4%   37.78Ki ± 5%  -12.15% (p=0.000 n=10)

         │    old     │                new                 │
         │ allocs/op  │ allocs/op   vs base                │
Create-8   166.0 ± 3%   146.0 ± 3%  -12.05% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-26 07:42:05 +00:00
Evgenii Stratonikov
dd3874eff1 [#447] pilorama: Add benchmark for create ops 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-26 07:42:05 +00:00
Dmitrii Stepanov
4d48377cec [#459] blobovniczatree: Fix get error 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 16:44:53 +03:00
Dmitrii Stepanov
b5d9f4a285 [#373] metrics: Add pilorama metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
e89fa110c7 [#373] metrics: Add metabase metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
af608da952 [#373] metrics: Add blobovnizca metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
56f320dd85 [#373] metrics: Add blobstor metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
16a142cd0c [#373] metrics: Add FSTree metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
d8ecc69d00 [#373] local storage: Pass parent ID 
This is required to add shard ID as metric label.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
d5aaec1107 [#373] pilorama: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
059e9e88a2 [#373] metabase: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
f54cc0b607 [#373] blobstor: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
8318d90ad0 [#373] blobovniczatree: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
3ae3c8dfdb [#373] fstree: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
a8526d45e9 [#373] blobovnizca: Add missed/fix tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
028d4a8058 [#373] blobovnicza: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
01a0c97760 [#453] engine: Set Disabled mode to deleted shard 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-20 12:04:07 +03:00
Evgenii Stratonikov
69df0d21c2 [#446] engine: Move to read-only on blobstor errors 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-16 14:53:32 +03:00
Evgenii Stratonikov
fe01781811 [#446] los: Wrap SSD errors in a separate type 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-16 14:53:31 +03:00
Evgenii Stratonikov
20b84f183a [#446] engine: Simplify logs for shard mode change 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-16 14:51:29 +03:00
Dmitrii Stepanov
4449006862 [#424] metrics: Use mode value as metric value for shard 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 18:26:19 +03:00
Dmitrii Stepanov
1b364d8cf4 [#424] metrics: Refactor engine metrics 
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:53:32 +03:00
Dmitrii Stepanov
85deb12f4d [#424] writecache: Drop metrics when close 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:52:40 +03:00
Dmitrii Stepanov
07f155ac77 [#424] metrics: Use labels for writecache methods and operations 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:52:40 +03:00
Dmitrii Stepanov
2541d319de [#266] pilorama: Allow to get current tree height 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-13 10:00:45 +00:00
Evgenii Stratonikov
41ab4d070e [#423] *: Use hrw.StringHash() where possible 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-13 07:18:25 +00:00
Anton Nikiforov
263c6fdc50 [#372] node: Add metrics for the error counter in the engine 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-06-07 13:04:47 +00:00
Dmitrii Stepanov
74578052f9 [#412] node: Replace tracing package 
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Dmitrii Stepanov
dbf41391b5 [#401] engine: Extend evacuation logs 
Add operation-tag to logger.
Log evacuation results.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-31 13:24:30 +03:00
Dmitrii Stepanov
3220c4df9f [#376] metrics: Add GC metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-31 10:22:12 +00:00
Alejandro Lopez
faca861451 [#411] Remove unnecessary pointers for sync objects 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-31 10:19:14 +00:00
Anton Nikiforov
8dcd06c587 [#394] node: Use Context in Blobovniczas.Iterate() 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-31 10:09:18 +00:00
Anton Nikiforov
365a7ca0f4 [#366] node: Stop GC once termination signal received 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-29 09:35:08 +03:00
Anton Nikiforov
802168c0c6 [#364] node: Stop flushing big object when termination signal received 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-05-26 16:46:58 +03:00
Dmitrii Stepanov
271a56c2ab [#395] metrics: Drop redundant metrics 
HistogramVec already has labeled counter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-26 13:39:37 +00:00
Evgenii Stratonikov
f2e5dead7e [#398] pilorama: Disallow applying same operations 
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
   overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-26 12:14:29 +00:00
Dmitrii Stepanov
20a489bdb5 [#393] gc: Use defer to mark handler done 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-26 12:14:02 +00:00
Dmitrii Stepanov
2613351008 [#387] gc: Cancel GC is change mode requested 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-25 09:38:16 +03:00
Dmitrii Stepanov
2ce43935f9 [#312] metrics: Add writecache metrcis 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Dmitrii Stepanov
d212d908b5 [#312] wc: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Dmitrii Stepanov
4503a61997 [#312] wc: Delete unused Iterate method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-24 10:18:39 +00:00
Evgenii Stratonikov
4b768fd115 [#381] *: Move to sync/atomic 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-23 08:18:01 +03:00
Evgenii Stratonikov
ff570847a4 [#381] go.mod: Update bbolt 
Adopt new `ForEachBucket` function where possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-22 11:49:14 +03:00
Evgenii Stratonikov
35c9b6b26d [#314] writecache: remove objects right after they are flushed 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy
bf79d06f03 [#314] writecache: Do not lose small objects on disk errors 
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-19 09:37:05 +00:00
Pavel Karpy
9e56592be3 [#314] writecache: Simplify background workers naming 
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-19 09:37:05 +00:00
Dmitrii Stepanov
e4889e06ba [#329] node: Make evacuate async 
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-19 08:43:52 +00:00
Dmitrii Stepanov
869fcbf591 [#332] gc: Fix expired complex object deletion 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-16 12:44:57 +00:00
Dmitrii Stepanov
ab07bad33d [#332] gc: Add complex object unit test 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-16 12:44:57 +00:00
Evgenii Stratonikov
c62025c836 [#321] metabase/test: execute tests in parallel 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
945454f60c [#321] engine/test: Execute tests in parallel 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
4578d00619 [#321] shard/test: Execute tests in parallel 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
d35e4c389f [#321] shard/test: Parallelize TestWriteCacheObjectLoss 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
969bfb603f [#321] shard/test: Parallelize TestShard_List 
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After:  0.109s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-12 09:45:03 +00:00
Dmitrii Stepanov
a181c9e434 [#332] gc: Add additional logging 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-10 17:34:40 +03:00
Alejandro Lopez
973af12854 [#327] tests: replace os.MkdirTemp with t.TempDir 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-05 11:24:51 +00:00
Evgenii Stratonikov
a1b4ba9980 [#209] compression: Do not store uncompressible data 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-03 16:28:29 +03:00
Dmitrii Stepanov
eca5c210dd [#299] evacuate: Add context cancel checks 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-02 15:55:30 +03:00
Evgenii Stratonikov
a9c4ba62c3 [#248] metabase: Remove storage group bucket 
Backwards compatible change, so no version increase.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 17:00:43 +03:00
Evgenii Stratonikov
58f1ba4b51 [#288] pilorama: Add missing operation in log 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
daa26f6e9b [#288] pilorama/test: Check operation order for TreeGetByPath() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
291f9e809a [#288] pilorama: Remove getMeta() wrapper 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
0045f1bcd4 [#288] pilorama: Use more descriptive names for memory tree 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Evgenii Stratonikov
f856ad7480 [#288] pilorama: Remove childMap from memory forest 
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 07:56:20 +00:00
Airat Arifullin
ada081dfd5 [#19] node: Make policier read shards concurrently 
* Introduce ListWithMultiCursor that simultaneously reads objects
  from different shards

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-27 11:11:47 +03:00
Evgenii Stratonikov
1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo() 
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:11 +03:00
Evgenii Stratonikov
f07e2d4812 [#285] lint: Fix revive/unused-parameter 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:04 +03:00
Pavel Karpy
09938a9841 Revert "[#262] meta: Do not return old expired objects" 
This reverts commit 3d23b087

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:55 +03:00
Pavel Karpy
e9461686b8 [#274] wc: Resolve possible deadlock 
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Pavel Karpy
6b6f33ed71 [#274] wc: Make wait groups work more explicit 
Do not run routine that calls `wg.Done()` inside, it is hard to read.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-20 19:22:54 +03:00
Airat Arifullin
6c90bb87f1 [#118] node: add ctx for unit tests for blobstor 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-19 18:09:33 +03:00
Pavel Karpy
3d23b08773 [#262] meta: Do not return old expired objects 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-19 13:37:38 +00:00
Airat Arifullin
13c8afcb02 [#118] node: add unit concurrent tests for blobstor 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-19 10:22:50 +00:00
Pavel Karpy
20cd080323 [#255] write-cache: Fix init race condition 
Do not use WC's internals in the initialization routines without mode
protection. WC should be able to change its mode even if the initialization
is not finished yet.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-19 09:29:07 +00:00
Pavel Karpy
262c9c2b93 [#256] blobovniczaTree: Make Exists test stable 
Corrupt and request _the same_ file.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-14 16:46:24 +03:00
Evgenii Stratonikov
8466894fdf [#250] control: remove DumpShard and RestoreShard RPC 
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 12:28:49 +00:00
Dmitrii Stepanov
6121b541b5 [#242] treesvc: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Dmitrii Stepanov
d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Alejandro Lopez
41eb3129ae [#139] Refactor blobovnicza exist test to not use chmod 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-04-14 12:16:14 +03:00
Evgenii Stratonikov
0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Evgenii Stratonikov
5d2affa5cd testutil: Fix linter warning 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-12 11:00:02 +03:00
Dmitrii Stepanov
0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Evgenii Stratonikov
c4865783fc [#236] blobstor/test: Prefill storage in parallel in read benchmark 
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:11 +03:00
Evgenii Stratonikov
6ad5c38225 [#236] testutil: Use random object id in RandObjGenerator 
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8            227425              4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8     --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8            --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:04 +03:00
Evgenii Stratonikov
c85a0bc866 [#236] blobstor/test: Reduce test descriptions 
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-11 20:52:04 +03:00
Evgenii Stratonikov
dbc3811ff4 [#191] engine: Allow to remove redundant object copies 
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-07 17:25:50 +00:00
Airat Arifullin
cb172e73a6 [#228] node: Use uber atomic package instead standard 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 15:37:27 +00:00
Dmitrii Stepanov
f32f61df87 [#203] pilorama: Refactor tests 
Do not pass 0 as channel capacity.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
8908798f59 [#203] node: Resolve unused vars 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
bab11492ad [#203] node: Resolve never used errors 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
9e2df4b7c7 [#203] node: Fix double imports 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Airat Arifullin
ab891517de [#116] node: Fix bug with extra generated files in TestReload 
* Create testNewEngine in engineWithShards without default opts

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-05 18:49:34 +03:00
Airat Arifullin
6f7b6a8813 [#116] node: Improve shard/engine construction in tests 
* Introduce testEngineWrapper that can be constructed with different options

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-05 14:36:40 +00:00
Dmitrii Stepanov
23575e1ac0 [#210] policier: Resolve contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Evgenii Stratonikov
9098d0eec0 [#211] engine: Unify shard mode checks for tree operations 
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-05 11:10:39 +00:00
Evgenii Stratonikov
760af6b912 [#211] fstree: Consider ENOSPC a logical error 
We already do this for file writing, however directory creation can also
fail.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-05 11:10:39 +00:00
Dmitrii Stepanov
1f1aed87be [#188] metabase: Refactor object inhume 
Resolve funlen linter for db.Inhume method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov
5a66db80c5 [#188] engine: Refactor shard evacuation 
Resolve funlen and gocognit linter for StorageEngine.Evacuate method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov
456bc097f7 [#188] engine: Refactor get range from engine 
Resolve funlen linter for StorageEngine.getRange method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov
3010ca2649 [#188] engine: Refactor get object from engine 
Resolve funlen linter for StorageEngine.get method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov
0739c36a3b [#188] metabase: Refactor object put to metabase 
Resolve funlen linter for db.put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov
8273a3dfb2 [#188] blobstor: Refactor blobstor test 
Resolve funlen linter for TestIterate function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov
594b5821ed [#188] blobstor: Refactor put data to blobovniczas 
Resolve funlen linter for Blobovniczas.Put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov
8e5a0dcf27 [#204] gc: Fix GC handlers start 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 06:48:27 +00:00
Dmitrii Stepanov
a7c79c773a [#168] node: Refactor node config 
Resolve containedctx linter for cfg

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-31 09:32:59 +03:00
Airat Arifullin
7be5a0fd79 [#189] node: Fix bug with sync/atomic package 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-03-30 14:50:43 +00:00
Dmitrii Stepanov
ab32067152 [#183] gc: Fix drop expired locked complex objects 
Do not delete bucket keys during iteration

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 15:33:42 +03:00
Dmitrii Stepanov
9f0bce5c15 [#183] gc: Fix drop expired locked simple objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 15:33:42 +03:00
Alejandro Lopez
341fe1688f [#139] test: Add test storage implementation 
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-03-29 14:28:49 +00:00
Pavel Karpy
97e201993b [#184] *: Unify done contexts handling 
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-29 10:34:54 +00:00
Airat Arifullin
221203beeb [#180] node: Refactor panics in unit test 
* Replace panics in unit tests by require.NoError and t.Fatalf

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-03-29 12:39:07 +03:00
aarifullin
34329d67ff [#86] node: Fix unit test and linter errors 
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
 2023-03-23 12:42:58 +03:00
Ayrat Arifullin
9808dec591 [#86] node: Move testing utils to one package 
Move testing utils from tests in local_object_storage package to
unified testutil package

Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
 2023-03-23 08:19:15 +00:00
Denis Kirillov
342e571d89 [#159] Add handle __SYSTEM__ sys attributes 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-22 17:35:20 +03:00
Pavel Karpy
da8da1c63a [#98] fstree: Do not fail iteration over just removed files 
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-22 07:26:27 +00:00
Evgenii Stratonikov
8014fdb21a [#156] metabase: Make freePotentialLocks() pass linter checks 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov
fb13902db9 [#156] shard: Make refillMetabase() pass linter checks 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov
47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply() 
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Dmitrii Stepanov
5059dcc19d [#145] shard-gc: Delete expired objects after locks 
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 11:31:08 +03:00
Dmitrii Stepanov
6c4a1699ef [#145] shard-gc: Expired locked unit test 
Added unit test that verifies that GC deletes expired
locked objects in one epoch.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 11:31:08 +03:00
Dmitrii Stepanov
481a1ca6f3 [#148] linter: Add gocognit linter 
Code with high cognitive complexity is hard intuitively to understand

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov
97c36ed3ec [#148] linter: Add funlen linter 
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov
2dc86058c3 [#148] memstore: Drop space line 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:52:39 +03:00
Pavel Karpy
f006f3b342 [#67] node: Make engine's IsLocked public 
It will allow reusing that method in expiration checks.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-16 16:20:45 +03:00
Alejandro Lopez
724debfdcd [#81] node: Add basic read/write benchmarks for substorages 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-03-15 16:37:04 +00:00
Evgenii Stratonikov
3e6fd4c611 [#82] pilorama: Allow to store last sync height 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 11:25:44 +00:00
Evgenii Stratonikov
861e9ab59a [#83] pre-commit: Add initial configuration 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 07:07:29 +00:00
Pavel Karpy
f1f3c80dbf [#32] node: Init write-cache asynchronously 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-09 11:07:33 +00:00
Pavel Karpy
381e363a8b [#32] node: Always close general components after testing 
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-09 11:07:33 +00:00
Alex Vanin
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Anton Nikiforov
e9f3c24229 [#65] Use strings.Cut instead of strings.Split* where possible 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-02-28 13:39:14 +03:00
Dmitrii Stepanov
6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods 
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-02-28 13:36:25 +03:00
Dmitrii Stepanov
c3a7039801 [TrueCloudLab/hrw#2] node: Optimize shard hash 
Compute shard hash only once

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-02-28 13:36:25 +03:00
Alejandro Lopez
cb5468abb8 [#66] node: Replace interface{} with any 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-02-21 16:47:07 +03:00
Pavel Karpy
337049b2ce [#56] node: Allow reading expired locked object 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-21 09:56:57 +03:00
Pavel Karpy
3beef10f89 [#61] node: Do not fetch missing objects 
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 14:47:38 +03:00
Evgenii Stratonikov
d1d123d180 [#2234] writecache: Fix possible panic in initFlushMarks 
In case we have many small objects in the write-cache, `indices` should
not be reused between iterations.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
315141dc2c [#2252] fstree: Allow concurrent writes 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Pavel Karpy
07ec51ea60 [#2244] node: Add object address to WC's operations 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 13:53:27 +03:00
Pavel Karpy
dbbbef9ddb [#2244] node: Update expired storage ID by WC 
Previously, node could get an "infinite" small object: it could be expired
and thus could not be flushed (update its storage ID) to metabase => could
not be marked as flushed => node never removes such object and repeat all
the cycle one more time. If object exists and is not marked with GC (meta
returns `ErrObjectIsExpired`, not `ObjectNotFound` and not
`ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt
transaction.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
5cb2c5ae62 [#2238] engine: Add test for component initialization failures 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
427fe276f2 [#2238] shard: Try closing all components 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
c53903ccd0 [#2238] engine: Make Open and Init similar 
1. Both could initialize shards in parallel.
2. Both should close shards after an error.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
e0309e398c [#2239] writecache: Fix possible deadlock 
LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction.
`View` transaction itself takes `mmapLock` [1], which is lifted after tx
finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`)

When we evict items from LRU cache mutex order is different:
first we take LRU mutex and then execute `Batch` which _does_ take
`mmapLock` in case we need to remap. Thus the deadlock.

[1] 8f4a7e1f92/db.go (L708)

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
58367e4df6 [#2232] pilorama: Merge in-queue batches 
To achieve high performance we must choose proper values for both
batch size and delay. For user operations we want to set low delay.
However it would prevent tree synchronization operations to form big
enough batches. For these operations, batching gives the most benefit
not only in terms of on-CPU execution cost, but also by speeding up
transaction persist (`fsync`).
In this commit we try merging batches that are already
_triggered_, but not yet _started to execute_. This way we can still
query batches for execution after the provided delay while also allowing
multiple formed batches to execute faster.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Pavel Karpy
40822adb51 [#2213] node: Do not return object expired object 
"Object is expired" means that object is presented in `meta` but it is not
`ObjectNotFound` error. Previous implementation made `shard` search for an
object without `meta` which was an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 13:53:27 +03:00