dstepanov-yadro/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
4244 commits 260 branches 26 tags 23 MiB
Commit graph

2808 commits

Author SHA1 Message Date
Dmitrii Stepanov
226e84d782 [#684] node: Add skipped objects count to evacuation result 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-07 12:17:11 +00:00
Dmitrii Stepanov
523fb3ca51 [#684] proto: Add skipped count to evacuation status response 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-07 12:17:11 +00:00
aarifullin
74c91eeef5 [#777] client: Refactor PrmContainerList, PrmObjectSearch usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-06 06:50:11 +00:00
Dmitrii Stepanov
20d6132f31 [#531] signSvc: Add SetMarshaledData method call 
To reduce memory allocations add `SetMarshaledData` method call
to return already marshalled data in next `StableMarshal` calls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-02 17:34:33 +03:00
Anton Nikiforov
a26483fc30 [#749] morph: Fix panic when closing morph client 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-11-01 10:48:10 +00:00
Dmitrii Stepanov
c80b46fad3 [#754] blobstor: Estimate compressability 
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-01 11:24:32 +03:00
Dmitrii Stepanov
05b508f79a [#772] proto: Fix file ending 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:04 +03:00
Dmitrii Stepanov
79088baa06 [#772] node: Apply gofumpt 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:03 +03:00
Evgenii Stratonikov
00aa6d9749 [#633] shard/test: Fix TestCounters() 
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov
261d281154 [#762] go.mod: Update sdk-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 11:22:29 +00:00
Dmitrii Stepanov
869518be0a [#728] writecache: Fix Badger writecache race. 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-30 18:36:41 +03:00
Dmitrii Stepanov
d4b6ebe7e7 [#725] writecache: Fix metric values 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-27 12:22:29 +03:00
Anton Nikiforov
121f5c4dd8 [#757] ir: Do not exclude node in maintenance mode from netmap 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-26 10:50:32 +03:00
Evgenii Stratonikov
9f7c2d8810 [#752] innerring: Simplify keyPosition() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
cddc58ace2 [#752] innerring: Optimize keyPosition() 
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
              │      old       │                 new                 │
              │     sec/op     │   sec/op     vs base                │
KeyPosition-8   2771.50n ± 10%   40.32n ± 4%  -98.55% (p=0.000 n=10)

              │     old      │                  new                  │
              │     B/op     │     B/op      vs base                 │
KeyPosition-8   1.531Ki ± 0%   0.000Ki ± 0%  -100.00% (p=0.000 n=10)

              │    old     │                new                 │
              │ allocs/op  │ allocs/op  vs base                 │
KeyPosition-8   21.00 ± 0%   0.00 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
0a9830564f [#752] morph: Adopt neo-go RPC statuses 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
6950312967 [#752] morph: Drop loop copy kludges 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
fe1acf9e9a [#750] morph: Remove deprecated channel use 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 07:57:05 +00:00
Anton Nikiforov
559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Anton Nikiforov
c0b86f2d93 [#642] writecache: Remove usage of close channel in badger 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Anton Nikiforov
b0cf100427 [#49] node: React on SIGHUP only when node in READY state 
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:55:29 +00:00
aarifullin
58b6224dd8 [#747] client: Refactor PrmObjectPutInit usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-10-20 11:55:40 +00:00
aarifullin
12b7cf2533 [#747] client: Refactor PrmObjectPutSingle usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-10-20 11:55:40 +00:00
Dmitrii Stepanov
dc4d27201b [#733] morph: Fix delete container signature check 
Committed invalid condition, it was just for debug.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-19 18:07:37 +03:00
Dmitrii Stepanov
189dbb01be [#733] frostfs-cli: Add control ir remove-container 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-19 16:22:18 +03:00
Dmitrii Stepanov
f2437f7ae9 [#734] shard: Fix Delete method 
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov
7e0c5a55de [#734] writecache: Fix flush 
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:17 +03:00
Anton Nikiforov
3a997d1207 [#680] metrics: Initialize log metrics together with services 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-11 17:08:03 +03:00
Anton Nikiforov
994f48f8bb [#680] metrics: Export log and morph with script 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-11 17:08:03 +03:00
Evgenii Stratonikov
aca11d7474 [#735] policer: Allow to provide metrics from the outside 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-11 15:14:13 +03:00
Ekaterina Lebedeva
5e229dc248 [#701] metrics: add metric to evaluate policer performance 
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-10-09 19:02:08 +00:00
Evgenii Stratonikov
4caa934eea [#729] containersvc: Remove load announcement 
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-09 19:01:13 +00:00
Dmitrii Stepanov
d07afd803c [#726] writecache: Fix small object flush for Badger 
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-06 11:32:50 +03:00
Dmitrii Stepanov
997ac7cd8d [#726] writecache: Fix small object flush for BBolt 
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-06 11:32:44 +03:00
Alexander Chuprov
c1e4130020 [#146] node: Add trace_id to logs 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-09-27 11:05:27 +03:00
a.arifullin
3889e829e6 [#667] writecache: Add logs for report error func in tests 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-09-14 17:02:54 +00:00
Evgenii Stratonikov
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch 
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-14 10:39:18 +03:00
Ekaterina Lebedeva
8a81af5a3b [#653] Add context parameter to Open functions 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-09-07 18:03:29 +03:00
Evgenii Stratonikov
a716db99db [#668] shard/test: Do not alter rootPath option 
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
36759f8434 [#668] shard/test: Properly check event processing 
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
39879fa868 [#668] shard/test: Add dontRelease options 
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
268adb79cb [#668] shard/test: Simplify shard construction 
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
429f941cda [#668] shard/test: Release shard in t.Cleanup() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
382eb8a485 [#668] shard/test: Disable GC where it is not needed 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
42696016de [#668] shard: Close stopChannel in GC 
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
bdecfbc1be [#668] shard/test: Move tests to the main package 
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters 
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00