carpawell/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
4399 commits 27 branches 26 tags 21 MiB
Commit graph

2915 commits

Author SHA1 Message Date
aarifullin
3534d6d05b [#794] objectsvc: Return accidentally removed acl checks for Head 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 17:13:58 +03:00
Airat Arifullin
66848d3288 [#770] cli: Add methods to work with APE rules via control svc 
* Add methods to frostfs-cli
* Implement rpc in control service

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
Airat Arifullin
8e11ef46b8 [#770] object: Introduce ape chain checker for object svc 
* Introduce Request type converted from RequestInfo type
  to implement policy-engine's Request interface
* Implement basic ape checker to check if a request is
  permitted to be performed
* Make put handlers use APE checker instead EACL

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
Airat Arifullin
5ec73fe8a0 [#770] node: Introduce ape chain source 
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
Airat Arifullin
3a2c319b87 [#770] control: Generate gRPC methods to manipulate APE chains 
* Define new types and gRPC methods to manipulate APE chains
  in control service.
* Stub gRPC handlers for the generated methods.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-08 13:34:03 +00:00
Dmitrii Stepanov
70ab1ebd54 [#763] metrics: Add container_objects_total metric 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov
9c98fa6152 [#763] metabase: Add container objects counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-08 12:30:57 +03:00
Dmitrii Stepanov
226e84d782 [#684] node: Add skipped objects count to evacuation result 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-07 12:17:11 +00:00
Dmitrii Stepanov
523fb3ca51 [#684] proto: Add skipped count to evacuation status response 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-07 12:17:11 +00:00
aarifullin
74c91eeef5 [#777] client: Refactor PrmContainerList, PrmObjectSearch usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-11-06 06:50:11 +00:00
Dmitrii Stepanov
20d6132f31 [#531] signSvc: Add SetMarshaledData method call 
To reduce memory allocations add `SetMarshaledData` method call
to return already marshalled data in next `StableMarshal` calls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-02 17:34:33 +03:00
Anton Nikiforov
a26483fc30 [#749] morph: Fix panic when closing morph client 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-11-01 10:48:10 +00:00
Dmitrii Stepanov
c80b46fad3 [#754] blobstor: Estimate compressability 
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-01 11:24:32 +03:00
Dmitrii Stepanov
05b508f79a [#772] proto: Fix file ending 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:04 +03:00
Dmitrii Stepanov
79088baa06 [#772] node: Apply gofumpt 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:03 +03:00
Evgenii Stratonikov
00aa6d9749 [#633] shard/test: Fix TestCounters() 
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 12:53:28 +00:00
Evgenii Stratonikov
261d281154 [#762] go.mod: Update sdk-go 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-31 11:22:29 +00:00
Dmitrii Stepanov
869518be0a [#728] writecache: Fix Badger writecache race. 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-30 18:36:41 +03:00
Dmitrii Stepanov
d4b6ebe7e7 [#725] writecache: Fix metric values 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-27 12:22:29 +03:00
Anton Nikiforov
121f5c4dd8 [#757] ir: Do not exclude node in maintenance mode from netmap 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-26 10:50:32 +03:00
Evgenii Stratonikov
9f7c2d8810 [#752] innerring: Simplify keyPosition() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
cddc58ace2 [#752] innerring: Optimize keyPosition() 
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
              │      old       │                 new                 │
              │     sec/op     │   sec/op     vs base                │
KeyPosition-8   2771.50n ± 10%   40.32n ± 4%  -98.55% (p=0.000 n=10)

              │     old      │                  new                  │
              │     B/op     │     B/op      vs base                 │
KeyPosition-8   1.531Ki ± 0%   0.000Ki ± 0%  -100.00% (p=0.000 n=10)

              │    old     │                new                 │
              │ allocs/op  │ allocs/op  vs base                 │
KeyPosition-8   21.00 ± 0%   0.00 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
0a9830564f [#752] morph: Adopt neo-go RPC statuses 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
6950312967 [#752] morph: Drop loop copy kludges 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 16:06:44 +03:00
Evgenii Stratonikov
fe1acf9e9a [#750] morph: Remove deprecated channel use 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-25 07:57:05 +00:00
Anton Nikiforov
559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Anton Nikiforov
c0b86f2d93 [#642] writecache: Remove usage of close channel in badger 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:57:50 +00:00
Anton Nikiforov
b0cf100427 [#49] node: React on SIGHUP only when node in READY state 
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-24 15:55:29 +00:00
aarifullin
58b6224dd8 [#747] client: Refactor PrmObjectPutInit usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-10-20 11:55:40 +00:00
aarifullin
12b7cf2533 [#747] client: Refactor PrmObjectPutSingle usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-10-20 11:55:40 +00:00
Dmitrii Stepanov
dc4d27201b [#733] morph: Fix delete container signature check 
Committed invalid condition, it was just for debug.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-19 18:07:37 +03:00
Dmitrii Stepanov
189dbb01be [#733] frostfs-cli: Add control ir remove-container 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-19 16:22:18 +03:00
Dmitrii Stepanov
f2437f7ae9 [#734] shard: Fix Delete method 
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:18 +03:00
Dmitrii Stepanov
7e0c5a55de [#734] writecache: Fix flush 
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-16 17:00:17 +03:00
Anton Nikiforov
3a997d1207 [#680] metrics: Initialize log metrics together with services 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-11 17:08:03 +03:00
Anton Nikiforov
994f48f8bb [#680] metrics: Export log and morph with script 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-10-11 17:08:03 +03:00
Evgenii Stratonikov
aca11d7474 [#735] policer: Allow to provide metrics from the outside 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-11 15:14:13 +03:00
Ekaterina Lebedeva
5e229dc248 [#701] metrics: add metric to evaluate policer performance 
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-10-09 19:02:08 +00:00
Evgenii Stratonikov
4caa934eea [#729] containersvc: Remove load announcement 
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-09 19:01:13 +00:00
Dmitrii Stepanov
d07afd803c [#726] writecache: Fix small object flush for Badger 
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-06 11:32:50 +03:00
Dmitrii Stepanov
997ac7cd8d [#726] writecache: Fix small object flush for BBolt 
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-06 11:32:44 +03:00
Alexander Chuprov
c1e4130020 [#146] node: Add trace_id to logs 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-09-27 11:05:27 +03:00
a.arifullin
3889e829e6 [#667] writecache: Add logs for report error func in tests 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-09-14 17:02:54 +00:00
Evgenii Stratonikov
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch 
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-14 10:39:18 +03:00
Ekaterina Lebedeva
8a81af5a3b [#653] Add context parameter to Open functions 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-09-07 18:03:29 +03:00
Evgenii Stratonikov
a716db99db [#668] shard/test: Do not alter rootPath option 
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
36759f8434 [#668] shard/test: Properly check event processing 
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
39879fa868 [#668] shard/test: Add dontRelease options 
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
268adb79cb [#668] shard/test: Simplify shard construction 
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
429f941cda [#668] shard/test: Release shard in t.Cleanup() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
382eb8a485 [#668] shard/test: Disable GC where it is not needed 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
42696016de [#668] shard: Close stopChannel in GC 
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
bdecfbc1be [#668] shard/test: Move tests to the main package 
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters 
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Evgenii Stratonikov
da8f384324 [#668] shard/test: Fix typo in existence 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-07 07:39:39 +00:00
Anton Nikiforov
aeeb8193d2 [#676] node: Fix header source creation when checking eacl 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-09-06 17:06:54 +03:00
Anton Nikiforov
88d50e4c77 [#656] policer: Add "bad" testcase 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-09-06 08:04:59 +00:00
Evgenii Stratonikov
a54b4472de [#674] network: Close connections on address updates 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-09-04 15:34:24 +03:00
Dmitrii Stepanov
7456c8556a [#536] blobovnicza: Add blobovniczatree DB cache 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-09-01 13:53:11 +03:00
Dmitrii Stepanov
c672f59ab8 [#536] blobovnicza: Drop cache 
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-09-01 13:51:26 +03:00
Evgenii Stratonikov
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-31 08:39:42 +00:00
Anton Nikiforov
4dff9555f1 [#568] writecache: Improve flushing scheme for badger 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-08-30 17:22:28 +00:00
Airat Arifullin
806cc13d9f [#658] client: Refactor PrmObjectGet/Head/Range usage 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-30 17:13:23 +00:00
Alejandro Lopez
1daef2ceeb [#660] writecache: Fix remaining addr2key uses 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-30 17:12:33 +00:00
Anton Nikiforov
fe5aa06a75 [#665] node: Bind length of copies number to number of replicas 
Allow to use one digit in copies number array for backward compatibility.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-08-30 17:11:55 +00:00
Dmitrii Stepanov
55b82e744b [#529] objectcore: Use common sender classifier 
Use common sender classifier for ACL service and format validator.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-29 10:33:06 +03:00
Dmitrii Stepanov
ae81d6660a [#529] objectcore: Fix object content validation 
There are old objects where the owner of the object
may not match the one who issued the token.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-29 10:33:06 +03:00
Dmitrii Stepanov
ab2614ec2d [#528] objectcore: Validate token issuer 
Add token issuer against object owner validation.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-29 10:10:10 +03:00
Airat Arifullin
4ea0df77d0 [#574] policer: Check if the container was really removed 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-28 14:21:38 +00:00
Airat Arifullin
554ff2c06b [#574] core: Extend Source interface with DeletionInfo method 
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-28 14:21:38 +00:00
Evgenii Stratonikov
9072772a09 [#649] shard/test: Increase GC remover interval 
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:10:25 +00:00
Evgenii Stratonikov
c4db8e7690 [#637] shard/test: Fix data race 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:10:25 +00:00
Evgenii Stratonikov
f8ba60aa0c [#648] objsvc/delete: Handle errors in Go style 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 09:45:35 +00:00
Evgenii Stratonikov
d2084ece41 [#648] objsvc/delete: Remove redundant logs 
We never propagate delete requests to the container node, because
tombstone broadcast is done via PUT. No need to pollute logs.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 09:45:35 +00:00
Evgenii Stratonikov
40b556fc19 [#647] objsvc/search: Improve testing coverage 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:40:01 +03:00
Evgenii Stratonikov
4db2cbc927 [#647] objsvc/search: Wrap in uniqueIDWriter during parameter setting 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:40:01 +03:00
Evgenii Stratonikov
966ad22abf [#647] objsvc/search: Simplify error handling 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:40:01 +03:00
Evgenii Stratonikov
56f841b022 [#647] objsvc/search: Remove TraverserGenerator wrapper 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:40:01 +03:00
Evgenii Stratonikov
ba58144de1 [#647] objsvc/search: Remove netmap.Source wrapper 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-25 10:40:01 +03:00
Evgenii Stratonikov
c9e3c9956e [#643] objsvc/put: Unify extraBroadcastEnabled usage 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-24 11:03:17 +03:00
Evgenii Stratonikov
facd3b2c4b [#643] objsvc/put: Unify placement iterators 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-24 11:03:17 +03:00
Evgenii Stratonikov
3fcf56f2fb [#643] objsvc/put: Copy config to distributedTarget 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-24 11:03:17 +03:00
Evgenii Stratonikov
96e690883f [#638] Unify test loggers 
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
322c1dc273 [#638] Use test.NewLogger() in tests 
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"

-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
02b03d9c4f [#638] logger: Remove sampling from test loggers 
Losing logs is always a bad idea, especially when we debug tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 11:21:05 +00:00
Evgenii Stratonikov
82cc453be9 [#xx] shard: Fix data race in metrics tests 
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-23 10:26:12 +00:00
Alejandro Lopez
345a1a69a2 [#635] Use internal key type when deleting from badger wc 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-22 10:53:19 +03:00
Airat Arifullin
23be3eb627 [#574] tree: Check if container is really removed 
* Use DeletionInfo method from morph client to check if
  the container has been really removed from neo-go

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-08-21 12:50:20 +03:00
Airat Arifullin
42fb6fb372 [#574] morph: Add DeletionInfo method for morph client 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-21 12:49:06 +03:00
Dmitrii Stepanov
354a92ea2c [#602] blobovnicza: Add leaf width implementation 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-21 10:27:32 +03:00
Alexander Chuprov
4d9a6c07fb [#618] core: Replace fmt.Sprintf with strconv.FormatUint 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-08-21 07:14:50 +00:00
Dmitrii Stepanov
f2811f8585 [#602] metrics: Add blobovnicza items counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-18 13:01:27 +03:00
Dmitrii Stepanov
c4e1d8eb07 [#602] node: Fix blobovnicza typos 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-18 11:14:10 +03:00
Dmitrii Stepanov
10e63537b2 [#602] metrics: Rename blobovnicza size metric 
`Size` is not size, but open db size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Dmitrii Stepanov
809e97626b [#602] blobovnicza: Fix size counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Dmitrii Stepanov
2e49d7ea7e [#602] blobovnicza: Init before using 
Fix blobovnicza size: after restart size metric resets.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-17 19:17:35 +00:00
Evgenii Stratonikov
f7042c5a6f [#609] Replace zaptest.NewLogger() with zap.L() 
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"

-zaptest.NewLogger(t)
+zap.L()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:51:10 +00:00
Evgenii Stratonikov
127c676786 [#607] *: Use keys.PublicKeys.Copy() where possible 
Semantic patch:
```
@@
var dst identifier
var src identifier
var keys identifier
@@
 import keys "github.com/nspcc-dev/neo-go/pkg/crypto/keys"

-dst := make(keys.PublicKeys, len(src))
-copy(dst, src)
+dst := src.Copy()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:50:02 +00:00
Evgenii Stratonikov
e604a3d749 [#607] *: Use zap.Stringer() where possible 
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
 f(
    ...,
-    zap.String(t, a.String()),
+    zap.Stringer(t, a),
    ...,
)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:50:02 +00:00
Evgenii Stratonikov
a8de37c8a2 [#607] *: Remove redundant if on error returns 
Semantic patch:
```
@@
@@
-if err != nil { return err }
-return nil
+return err
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 16:50:02 +00:00
Evgenii Stratonikov
5a51b78946 [#620] object: Send status response for server-side streams 
Previously status responses were wrapped in the gRPC error and thus
couldn't be correctly handled on client.

Introduced in c2617baf63, thanks @ale64bit for having found.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-17 12:52:38 +00:00
Anton Nikiforov
6407bb5bd1 [#619] node: Fix object put when copies numbers contains only zeros 
In this case object should placement according to replicas.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-08-17 11:31:34 +03:00
Evgenii Stratonikov
5335e7089e [#615] pilorama: Speedup TestForest_ApplyRandom() 
Some of our pilorama tests fail on CI.
The reasons are not obvious, but one possible improvement
is using `WithNoSync` option for these. It should have much effect,
because we are writing on the tmpfs, but doesn't hurt anyway.

If I replace `t.TempDir()` with a local directory, test execution time
goes down from 5s (sync) to 0.4s (nosync), which is the same time as
with `t.TempDir()`. Maybe we have some strange CI configuration.

```
panic: test timed out after 10m0s
running tests:
	TestForest_ApplyRandom (8m22s)
	TestForest_ApplyRandom/bbolt (8m21s)
...
goroutine 170 [syscall]:
syscall.Syscall(0xc000100000?, 0xc00047b758?, 0x6aff9a?, 0xc00041c1b0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:69 +0x27
syscall.Fdatasync(0x9e35c0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/zsyscall_linux_amd64.go:418 +0x2a
go.etcd.io/bbolt.fdatasync(0xc000189000?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-16 19:09:08 +00:00
Dmitrii Stepanov
2efe9cc1be [#585] writecache: Fix DB counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
58c8722c81 [#585] fstree: Add optional file counter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
baad49990c [#585] fstree: Return logical error if object deleted 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
0c52186572 [#585] fstree: Remove unused method 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Dmitrii Stepanov
eec97d177e [#585] writecache: Count items periodically 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-16 14:47:44 +03:00
Evgenii Stratonikov
d15199c5d8 [#596] engine: Consider context errors as logical 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-16 10:39:41 +03:00
Alejandro Lopez
88b6755c5e [#598] Fix use-after-close bug in badger writecache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-15 07:04:06 +00:00
Alejandro Lopez
ae8be495c8 [#xx] Avoid manual management of files in tests 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-14 14:01:39 +03:00
Alejandro Lopez
376f03a445 [#598] Hold mode mutex when setting mode 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-14 07:12:27 +00:00
Alexander Chuprov
21800e9fcc [#162] core: Move literals to constants 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-08-11 15:48:42 +03:00
Evgenii Stratonikov
b44a8dd46c [#597] *: Fix linter warnings 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-11 09:55:51 +03:00
Evgenii Stratonikov
e084c47bd6 [#248] innerring: Remove audit from tests 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-10 16:38:43 +00:00
Evgenii Stratonikov
779da6ec35 [#248] morph: Remove audit contract name 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-10 16:38:43 +00:00
Evgenii Stratonikov
55ce4dc075 [#248] morph: Remove obsolete network parameters 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-10 16:38:43 +00:00
Anton Nikiforov
4ad0ebb32f [#565] Add metrics for current GRPC endpoint status 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-08-10 16:37:49 +00:00
Evgenii Stratonikov
c3e23a1448 [#578] gendoc: Allow to override flags 
The command is used in multiple places across the whole FrostFS
ecosystem. While we want to have uniform interfaces everywhere,
sometimes we can't: already defined global flags can be harder to change
because of our obligations to the users. Cobra framework doesn't allow
conflicting flags (we can have global ones), so allow to override them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-10 11:10:11 +00:00
Evgenii Stratonikov
6bcba27757 [#578] gendoc: Remove flag shorthands 
Make it harder to encounter conflicts in already existing commands.
Because the command is executed once, I don't think the usability is
worse.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-10 11:10:11 +00:00
Evgenii Stratonikov
dca31e8888 [#578] gendoc: Add date to man pages 
```
HISTORY
       8-Aug-2023 Auto generated by spf13/cobra
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-10 11:10:11 +00:00
Evgenii Stratonikov
b02a1a34c1 [#578] gendoc: Allow to customize man pages 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-10 11:10:11 +00:00
Evgenii Stratonikov
082370583f [#578] gendoc: Replace NSPCC with FrostFS 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-10 11:10:11 +00:00
Alejandro Lopez
8f994163ee [#586] Fix writecache benchmarks and refactor hacky NeedsCompression 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-10 08:05:18 +00:00
Alejandro Lopez
023b90342c [#584] Disable compression in badger writecache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-10 10:14:06 +03:00
Evgenii Stratonikov
d641cba2fc [#587] Do not use math/rand.Read 
Fix staticcheck warnings after go1.20 update.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-09 16:02:44 +03:00
Alejandro Lopez
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 10:26:53 +00:00
Alejandro Lopez
ae322e9f73 [#576] Set SyncWrites for badger writecache by default 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 11:28:26 +03:00
Anton Nikiforov
8d589314b5 [#560] node: Fix Put in multi REP with intersecting sets of nodes 
Once the node was processed it skipped, at the step of forming
result in case when all nodes skipped, because processed for
previous REP, service mark the whole request as incomplete.

Example of policies which are unblocked:
- REP 1 REP 1 CBF 1
- REP 4 IN X REP 4 IN Y
  CBF 4
  SELECT 2 FROM FX AS X SELECT 2 FROM FY AS Y
  FILTER Country EQ Russia OR Country EQ Sweden OR Country EQ Finland AS FY
  FILTER Price GE 0 AS FX

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-08-08 10:22:53 +00:00
Alejandro Lopez
7da4306e38 [#575] writecache: Fix log level for badger writecache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-08 08:16:36 +00:00
Dmitrii Stepanov
0e697266c3 [#563] writecache: Fix metrics and bolt delete 
Estimate cache size after delete objects to update metric.
Update counters on small object deletion.
Do not count bbolt DB file as FSTree object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-08-07 12:53:28 +00:00
Alejandro Lopez
1a0cb0f34a [#421] Try using badger for the write-cache 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-07 08:16:57 +00:00
Alejandro Lopez
65c72f3e0b [#559] Remove manual path handling in fstree tests 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-03 10:03:41 +03:00
Evgenii Stratonikov
1e8b4b8a17 [#557] services: Regenerate stable marshalers 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-08-02 13:37:40 +00:00
Airat Arifullin
b3695411d9 [#553] eacl: Fix bug with casting to ObjectAccessDenied error 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-08-02 07:22:48 +00:00
Evgenii Stratonikov
35dc64bd7b [#547] metabase: Fix datarace in tests 
Quite an old one bf9e938a3b.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-31 15:35:15 +00:00
Evgenii Stratonikov
05ac9e3637 [#547] objectsvc: Work with traversal struct from a single thread 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-31 15:35:15 +00:00
Evgenii Stratonikov
7b0fdf0202 [#533] services: Assume API supports status codes 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-31 15:35:15 +00:00
Evgenii Stratonikov
ec8b4fdc48 [#541] writecache/test: Close writecache on exit 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-29 10:57:25 +00:00
Evgenii Stratonikov
ad5f527bd3 [#541] writecache/test: Remove initWC() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-29 10:57:25 +00:00
Dmitrii Stepanov
ea32913430 [#543] putsvc: Fix PutSingle implementation 
Add Lock and Delete handlers to local PutSingle.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-07-28 12:09:41 +00:00
Dmitrii Stepanov
99bb488ebd [#539] getsvc: Write payload direct to out stream 
To reduce memory allocations.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-07-27 17:02:08 +03:00
Dmitrii Stepanov
286242cad0 [#539] getsvc: Use buffer to assemble object 
To reduce memory consumption.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-07-27 17:02:08 +03:00
Evgenii Stratonikov
5ff82ff04f [#6] services/object: Simplify local/remote targets 
We do not use the return result from Close() and we always execute both
methods in succession. It makes sense to unite them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-21 18:39:12 +03:00
Evgenii Stratonikov
448b48287c [#6] services/util: Do not panic in sign function 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-21 18:39:12 +03:00
Evgenii Stratonikov
c2617baf63 [#6] services/util: Remove remaining stream wrappers 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-21 18:39:09 +03:00
Evgenii Stratonikov
372160d048 [#6] services/util: Remove SignService.HandleUnaryRequest 
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-21 18:36:08 +03:00