dstepanov-yadro/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
4257 commits 268 branches 26 tags 31 MiB
Commit graph

2819 commits

Author SHA1 Message Date
Dmitrii Stepanov
0f75e48138 [#796] policer: Fix tombstone objects replication 
Tombstone objects must be replicated to all container nodes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-09 13:39:33 +03:00
Dmitrii Stepanov
7cdae4f660 [#792] proto: Regenerate with fixed version 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-09 10:09:13 +00:00
Dmitrii Stepanov
9133b4389e [#788] objectsvc: Fix formatting (gofumpt) 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-09 10:27:32 +03:00
Dmitrii Stepanov
1b22801eed [#788] engine: Fix flaky tests 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-09 10:25:46 +03:00
aarifullin
3534d6d05b [#794] objectsvc: Return accidentally removed acl checks for Head 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 17:13:58 +03:00
Airat Arifullin
66848d3288 [#770] cli: Add methods to work with APE rules via control svc 
* Add methods to frostfs-cli
* Implement rpc in control service

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
Airat Arifullin
8e11ef46b8 [#770] object: Introduce ape chain checker for object svc 
* Introduce Request type converted from RequestInfo type
  to implement policy-engine's Request interface
* Implement basic ape checker to check if a request is
  permitted to be performed
* Make put handlers use APE checker instead EACL

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
Airat Arifullin
5ec73fe8a0 [#770] node: Introduce ape chain source 
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
Airat Arifullin
3a2c319b87 [#770] control: Generate gRPC methods to manipulate APE chains 
* Define new types and gRPC methods to manipulate APE chains
  in control service.
* Stub gRPC handlers for the generated methods.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
Dmitrii Stepanov
70ab1ebd54 [#763] metrics: Add container_objects_total metric 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov
9c98fa6152 [#763] metabase: Add container objects counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov
226e84d782 [#684] node: Add skipped objects count to evacuation result 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-07 12:17:11 +00:00
Dmitrii Stepanov
523fb3ca51 [#684] proto: Add skipped count to evacuation status response 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-07 12:17:11 +00:00
aarifullin
74c91eeef5 [#777] client: Refactor PrmContainerList, PrmObjectSearch usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-06 06:50:11 +00:00
Dmitrii Stepanov
20d6132f31 [#531] signSvc: Add SetMarshaledData method call 
To reduce memory allocations add `SetMarshaledData` method call
to return already marshalled data in next `StableMarshal` calls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-02 17:34:33 +03:00
Anton Nikiforov
a26483fc30 [#749] morph: Fix panic when closing morph client 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-11-01 10:48:10 +00:00
Dmitrii Stepanov
c80b46fad3 [#754] blobstor: Estimate compressability 
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-01 11:24:32 +03:00
Dmitrii Stepanov
05b508f79a [#772] proto: Fix file ending 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:04 +03:00
Dmitrii Stepanov
79088baa06 [#772] node: Apply gofumpt 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:03 +03:00
Evgenii Stratonikov
00aa6d9749 [#633] shard/test: Fix TestCounters() 
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov
261d281154 [#762] go.mod: Update sdk-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 11:22:29 +00:00
Dmitrii Stepanov
869518be0a [#728] writecache: Fix Badger writecache race. 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-30 18:36:41 +03:00
Dmitrii Stepanov
d4b6ebe7e7 [#725] writecache: Fix metric values 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-27 12:22:29 +03:00
Anton Nikiforov
121f5c4dd8 [#757] ir: Do not exclude node in maintenance mode from netmap 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-26 10:50:32 +03:00
Evgenii Stratonikov
9f7c2d8810 [#752] innerring: Simplify keyPosition() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
cddc58ace2 [#752] innerring: Optimize keyPosition() 
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
              │      old       │                 new                 │
              │     sec/op     │   sec/op     vs base                │
KeyPosition-8   2771.50n ± 10%   40.32n ± 4%  -98.55% (p=0.000 n=10)

              │     old      │                  new                  │
              │     B/op     │     B/op      vs base                 │
KeyPosition-8   1.531Ki ± 0%   0.000Ki ± 0%  -100.00% (p=0.000 n=10)

              │    old     │                new                 │
              │ allocs/op  │ allocs/op  vs base                 │
KeyPosition-8   21.00 ± 0%   0.00 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
0a9830564f [#752] morph: Adopt neo-go RPC statuses 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
6950312967 [#752] morph: Drop loop copy kludges 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
fe1acf9e9a [#750] morph: Remove deprecated channel use 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 07:57:05 +00:00
Anton Nikiforov
559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Anton Nikiforov
c0b86f2d93 [#642] writecache: Remove usage of close channel in badger 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Anton Nikiforov
b0cf100427 [#49] node: React on SIGHUP only when node in READY state 
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:55:29 +00:00
aarifullin
58b6224dd8 [#747] client: Refactor PrmObjectPutInit usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-10-20 11:55:40 +00:00
aarifullin
12b7cf2533 [#747] client: Refactor PrmObjectPutSingle usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-10-20 11:55:40 +00:00
Dmitrii Stepanov
dc4d27201b [#733] morph: Fix delete container signature check 
Committed invalid condition, it was just for debug.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-19 18:07:37 +03:00
Dmitrii Stepanov
189dbb01be [#733] frostfs-cli: Add control ir remove-container 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-19 16:22:18 +03:00
Dmitrii Stepanov
f2437f7ae9 [#734] shard: Fix Delete method 
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov
7e0c5a55de [#734] writecache: Fix flush 
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:17 +03:00
Anton Nikiforov
3a997d1207 [#680] metrics: Initialize log metrics together with services 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-11 17:08:03 +03:00
Anton Nikiforov
994f48f8bb [#680] metrics: Export log and morph with script 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-11 17:08:03 +03:00
Evgenii Stratonikov
aca11d7474 [#735] policer: Allow to provide metrics from the outside 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-11 15:14:13 +03:00
Ekaterina Lebedeva
5e229dc248 [#701] metrics: add metric to evaluate policer performance 
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-10-09 19:02:08 +00:00
Evgenii Stratonikov
4caa934eea [#729] containersvc: Remove load announcement 
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-09 19:01:13 +00:00
Dmitrii Stepanov
d07afd803c [#726] writecache: Fix small object flush for Badger 
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-06 11:32:50 +03:00
Dmitrii Stepanov
997ac7cd8d [#726] writecache: Fix small object flush for BBolt 
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-06 11:32:44 +03:00
Alexander Chuprov
c1e4130020 [#146] node: Add trace_id to logs 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-09-27 11:05:27 +03:00
a.arifullin
3889e829e6 [#667] writecache: Add logs for report error func in tests 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-09-14 17:02:54 +00:00
Evgenii Stratonikov
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch 
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-14 10:39:18 +03:00