Dmitrii Stepanov
f2437f7ae9
[ #734 ] shard: Fix Delete method
...
DCO action / DCO (pull_request) Successful in 1m36s
Details
Build / Build Components (1.20) (pull_request) Successful in 3m46s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m16s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m42s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m35s
Details
Tests and linters / Tests with -race (pull_request) Successful in 6m33s
Details
Build / Build Components (1.21) (pull_request) Successful in 13m7s
Details
Tests and linters / Lint (pull_request) Successful in 19m14s
Details
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
Alexander Chuprov
c1e4130020
[ #146 ] node: Add trace_id to logs
...
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
Details
DCO action / DCO (pull_request) Successful in 3m36s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Details
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Details
Tests and linters / Lint (pull_request) Successful in 5m2s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Details
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Details
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
Ekaterina Lebedeva
8a81af5a3b
[ #653 ] Add context parameter to Open functions
...
DCO action / DCO (pull_request) Successful in 1m38s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m22s
Details
Build / Build Components (1.21) (pull_request) Successful in 4m25s
Details
Vulncheck / Vulncheck (pull_request) Successful in 4m56s
Details
Tests and linters / Lint (pull_request) Successful in 6m1s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s
Details
Tests and linters / Staticcheck (pull_request) Successful in 8m1s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s
Details
Tests and linters / Tests with -race (pull_request) Successful in 8m32s
Details
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
Evgenii Stratonikov
a716db99db
[ #668 ] shard/test: Do not alter rootPath option
...
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
36759f8434
[ #668 ] shard/test: Properly check event processing
...
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
39879fa868
[ #668 ] shard/test: Add dontRelease options
...
Most of the time we would like to close shard with minor exceptions.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
c661ba1312
[ #668 ] shard/test: Use sane defaults in the test constructor
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
268adb79cb
[ #668 ] shard/test: Simplify shard construction
...
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
429f941cda
[ #668 ] shard/test: Release shard in t.Cleanup()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
382eb8a485
[ #668 ] shard/test: Disable GC where it is not needed
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
42696016de
[ #668 ] shard: Close stopChannel in GC
...
It is done once, but now we could read it from multiple places.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
bdecfbc1be
[ #668 ] shard/test: Move tests to the main package
...
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"
-shard.e
+e
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
aa23c6a83a
[ #668 ] shard/test: Remove subtest from TestCounters
...
Otherwise, individual tests cannot be run.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
da8f384324
[ #668 ] shard/test: Fix typo in `existence`
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
b9b86d2ec8
[ #666 ] shard/test: Fix data race in metrics tests
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-31 08:39:42 +00:00
Evgenii Stratonikov
9072772a09
[ #649 ] shard/test: Increase GC remover interval
...
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
Evgenii Stratonikov
c4db8e7690
[ #637 ] shard/test: Fix data race
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
Evgenii Stratonikov
96e690883f
[ #638 ] Unify test loggers
...
In some places we have debug=false, in others debug=true.
Let's be consistent.
Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
322c1dc273
[ #638 ] Use test.NewLogger() in tests
...
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"
-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
82cc453be9
[#xx] shard: Fix data race in metrics tests
...
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 10:26:12 +00:00
Alexander Chuprov
4d9a6c07fb
[ #618 ] core: Replace fmt.Sprintf with strconv.FormatUint
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00
Evgenii Stratonikov
f7042c5a6f
[ #609 ] Replace zaptest.NewLogger() with zap.L()
...
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"
-zaptest.NewLogger(t)
+zap.L()
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:51:10 +00:00
Evgenii Stratonikov
e604a3d749
[ #607 ] *: Use zap.Stringer() where possible
...
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
f(
...,
- zap.String(t, a.String()),
+ zap.Stringer(t, a),
...,
)
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
Alexander Chuprov
21800e9fcc
[ #162 ] core: Move literals to constants
...
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
Evgenii Stratonikov
d641cba2fc
[ #587 ] Do not use math/rand.Read
...
Build / Build Components (1.20) (pull_request) Failing after 3s
Details
Build / Build Components (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Lint (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.20) (pull_request) Failing after 3s
Details
Tests and linters / Tests (1.21) (pull_request) Failing after 3s
Details
Tests and linters / Tests with -race (pull_request) Failing after 5s
Details
Tests and linters / Staticcheck (pull_request) Failing after 8s
Details
Vulncheck / Vulncheck (pull_request) Failing after 2s
Details
Fix staticcheck warnings after go1.20 update.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
Alejandro Lopez
5b7e4a51b7
[ #481 ] Update frostfs-sdk-go and error pointer receivers
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 10:26:53 +00:00
Alejandro Lopez
1a0cb0f34a
[ #421 ] Try using badger for the write-cache
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-07 08:16:57 +00:00
Dmitrii Stepanov
b4e72a2dfd
[ #335 ] treesvc: Sort nodes by Filename in GetSubTree
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
Alexander Chuprov
033eaf77e1
[ #496 ] node: Fix linter importas
...
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Details
Build / Build Components (1.19) (pull_request) Successful in 4m1s
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Details
Tests and linters / Lint (pull_request) Successful in 14m40s
Details
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
Dmitrii Stepanov
cac4ed93d6
[ #428 ] engine: Add low_mem config parameter
...
Concurrent initialization in case of the metabase resync leads to
high memory consumption and potential OOM.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
Dmitrii Stepanov
16a142cd0c
[ #373 ] metrics: Add FSTree metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
d8ecc69d00
[ #373 ] local storage: Pass parent ID
...
This is required to add shard ID as metric label.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
059e9e88a2
[ #373 ] metabase: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
Dmitrii Stepanov
01a0c97760
[ #453 ] engine: Set Disabled mode to deleted shard
...
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-20 12:04:07 +03:00
Dmitrii Stepanov
4449006862
[ #424 ] metrics: Use mode value as metric value for shard
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
Dmitrii Stepanov
1b364d8cf4
[ #424 ] metrics: Refactor engine metrics
...
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
Dmitrii Stepanov
2541d319de
[ #266 ] pilorama: Allow to get current tree height
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
Anton Nikiforov
263c6fdc50
[ #372 ] node: Add metrics for the error counter in the engine
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-07 13:04:47 +00:00
Dmitrii Stepanov
74578052f9
[ #412 ] node: Replace tracing package
...
Use observability module.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
Dmitrii Stepanov
3220c4df9f
[ #376 ] metrics: Add GC metrics
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:22:12 +00:00
Anton Nikiforov
365a7ca0f4
[ #366 ] node: Stop GC once termination signal received
...
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-29 09:35:08 +03:00
Anton Nikiforov
802168c0c6
[ #364 ] node: Stop flushing big object when termination signal received
...
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
Dmitrii Stepanov
20a489bdb5
[ #393 ] gc: Use defer to mark handler done
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
Dmitrii Stepanov
2613351008
[ #387 ] gc: Cancel GC is change mode requested
...
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 09:38:16 +03:00
Dmitrii Stepanov
2ce43935f9
[ #312 ] metrics: Add writecache metrcis
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
Evgenii Stratonikov
4b768fd115
[ #381 ] *: Move to sync/atomic
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-23 08:18:01 +03:00
Dmitrii Stepanov
e4889e06ba
[ #329 ] node: Make evacuate async
...
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
Dmitrii Stepanov
869fcbf591
[ #332 ] gc: Fix expired complex object deletion
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
Dmitrii Stepanov
ab07bad33d
[ #332 ] gc: Add complex object unit test
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
Evgenii Stratonikov
4578d00619
[ #321 ] shard/test: Execute tests in parallel
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
d35e4c389f
[ #321 ] shard/test: Parallelize TestWriteCacheObjectLoss
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
Evgenii Stratonikov
969bfb603f
[ #321 ] shard/test: Parallelize TestShard_List
...
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After: 0.109s
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
Dmitrii Stepanov
a181c9e434
[ #332 ] gc: Add additional logging
...
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 17:34:40 +03:00
Evgenii Stratonikov
8466894fdf
[ #250 ] control: remove `DumpShard` and `RestoreShard` RPC
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 12:28:49 +00:00
Dmitrii Stepanov
6121b541b5
[ #242 ] treesvc: Add tracing spans
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
Dmitrii Stepanov
d62c6e4ce6
[ #242 ] node: Add tracing spans
...
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
Evgenii Stratonikov
0e31c12e63
[ #240 ] logs: Move log messages to constants
...
Drop duplicate entities.
Format entities.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
Dmitrii Stepanov
0920d848d0
[ #135 ] get-object: Add tracing spans
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
Airat Arifullin
cb172e73a6
[ #228 ] node: Use uber atomic package instead standard
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 15:37:27 +00:00
Dmitrii Stepanov
8908798f59
[ #203 ] node: Resolve unused vars
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
9e2df4b7c7
[ #203 ] node: Fix double imports
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
Dmitrii Stepanov
23575e1ac0
[ #210 ] policier: Resolve contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
Evgenii Stratonikov
9098d0eec0
[ #211 ] engine: Unify shard mode checks for tree operations
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
Dmitrii Stepanov
8e5a0dcf27
[ #204 ] gc: Fix GC handlers start
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 06:48:27 +00:00
Dmitrii Stepanov
a7c79c773a
[ #168 ] node: Refactor node config
...
Resolve containedctx linter for cfg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
Dmitrii Stepanov
9f0bce5c15
[ #183 ] gc: Fix drop expired locked simple objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
Alejandro Lopez
341fe1688f
[ #139 ] test: Add test storage implementation
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-29 14:28:49 +00:00
aarifullin
34329d67ff
[ #86 ] node: Fix unit test and linter errors
...
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 12:42:58 +03:00
Airat Arifullin
9808dec591
[ #86 ] node: Move testing utils to one package
...
ci/woodpecker/push/pre-commit Pipeline failed
Details
Move testing utils from tests in local_object_storage package to
unified testutil package
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 08:19:15 +00:00
Evgenii Stratonikov
fb13902db9
[ #156 ] shard: Make refillMetabase() pass linter checks
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
Evgenii Stratonikov
47e8c5bf23
[ #156 ] pilorama: Remove CIDDescriptor from TreeApply()
...
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
only used in controlled environments.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
Dmitrii Stepanov
5059dcc19d
[ #145 ] shard-gc: Delete expired objects after locks
...
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
Dmitrii Stepanov
6c4a1699ef
[ #145 ] shard-gc: Expired locked unit test
...
Added unit test that verifies that GC deletes expired
locked objects in one epoch.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
Dmitrii Stepanov
97c36ed3ec
[ #148 ] linter: Add funlen linter
...
Long functions are hard to understand and source of errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
Evgenii Stratonikov
3e6fd4c611
[ #82 ] pilorama: Allow to store last sync height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
Alexey Vanin
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
Pavel Karpy
3beef10f89
[ #61 ] node: Do not fetch missing objects
...
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 14:47:38 +03:00
Pavel Karpy
07ec51ea60
[ #2244 ] node: Add object address to WC's operations
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Evgenii Stratonikov
427fe276f2
[ #2238 ] shard: Try closing all components
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Artem Tataurov
362f24953a
[ #47 ] shard: Switch container size metric from physical to logical capacity
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-17 12:03:42 +03:00
Artem Tataurov
ab21d90cfb
[ #1794 ] shard: Add increasing case for the payload size metric
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-09 13:30:23 +03:00
Stanislav Bogatyrev
cb016d53a6
[ #1 ] Fix comments and error messages
...
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
Pavel Karpy
89a0266f5e
[ #1794 ] metrics: Track physical object capacity per shard
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-01-26 20:06:28 +03:00
Evgenii Stratonikov
9513f163aa
[ #2116 ] metrics: Track physical object capacity in the container
...
Currently we track based on `PayloadSize`, because it is already stored
in the metabase and it is easier to calculate without slowing down the
whole system.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-01-26 20:06:28 +03:00
Evgenii Stratonikov
d65a95a2c6
[ #28 ] pilorama: Remove `LogMove` struct
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
c72576e72f
[ #2208 ] engine: Log time-consuming shard operations
...
Currently the only way to tell whether `evacuate/set-mode` is finished
is to set a very big timeout and _hope_ that the operation will finish.
In this commit we add INFO logs for such operations which should
simplify the life of an administrator.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
64a5294b27
[ #2200 ] shard: Do not fetch big objects from blobovniczas
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
91757329ae
[ #2200 ] shard: Fix blobstor obj fetching
...
In the previous implementation any non-nil error that preceded object
fetching from blobstor led to iterating over every storage (in other words,
no storage ID information was taken into account). Now storage ID is
skipped only if metabase (storage ID source) returns any error.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
6451f019d2
[ #2203 ] shard: Do not panic in `Close` after unsuccessful `Init`
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
b4e90cdf51
[ #2165 ] pilorama: Optimize `TreeApply` when used for synchronization
...
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21717262ec
[ #2016 ] shard: Check meta first on `Get`
...
`meta` should prevent returning removed objects (`GCMark` and `TS` relations
are `meta` abstractions).
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
74ec71446f
[ #2167 ] shard: Do not use write-cache by default in `Head`
...
Both `meta` and `write-cache` are expected to have a fast underlying disk,
so it does not seem like an optimisation. Moreover, `write-cache`'s `Head`
is a `Get` with payload cutting, it _must_ use more memory for no reason
(`meta` was created for such requests). Also, `write-cache` does not allow
performing any "meta" relations checks (such as locking, tombstoning).
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
eea2892109
[ #1956 ] node: Lock shard's mode on its methods switch
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Anton Nikiforov
edb1428248
[ #2022 ] Add metric `readonly` to get shards mode
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a
Move to frostfs-node
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Pavel Karpy
634792077e
[ #1502 ] node: Store lock object on every container node
...
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
34e8d2ba56
[ #1502 ] shard: Add `IsLocked` method
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d65604ad30
[ #1985 ] blobstor: Allow to report multiple errors to caller
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
b0e94b6a6b
[ #1906 ] writecache: Do not require `read-only` mode in `Flush`
...
It was needed before we started to flush during transition to
`degraded` mode. Now it is confusing.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d8d3588e1b
[ #1996 ] engine: Always select proper shard for a tree
...
Currently there is a possibility for modifying operations to fail
because of I/O errors and a new tree to be created on another shard.
This commit adds existence check for modifying operations.
Read operations remain as they are, not to slow things.
`TreeDrop` is an exception, because this is a tree removal and trying
multiple shards is not an unwanted behaviour.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-03 15:29:23 +03:00