Commit graph

2992 commits

Author SHA1 Message Date
29550fe600 [#838] shard: Refactor updateMetrics method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
b2c63e57ba [#651] engine/test: Speedup StorageEngine_Inhume
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
445ebcc0e7 [#651] shard/test: Speedup Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
2302e5d342 [#651] shard/test: Refactor Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
26e4f7005c [#741] treesvc: Refactor tree sync
Fix linter issues.
Add error logging.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
b21be1abdd [#741] treesvc: Do not update sync height if some node is unavailable
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
b215817e14 [#741] treesvc: Remove unused height variables
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
c516c7c5f4 [#821] node: Pass user.ID by value
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-23 10:21:07 +03:00
07390ad4e3 [#715] node: Unify config parameter names
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 17:13:50 +03:00
8d18fa159e [#667] writecache: Fix flush test
Allow to disable background flusher for testing purposes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 17:45:43 +03:00
02454df14a [#809] client: Refactor PrmInit, PrmDial usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-17 13:37:03 +00:00
76ff26039c [#96] node: Drop neo-go's slices package
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 13:24:04 +03:00
47286ebf32 [#805] pilorama: Fix TreeDrop
* If treeID is empty then deleting buckets for cursor may get
  invalidated. So, buckets should be gathered before deleting.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-17 10:21:35 +00:00
29fe8c41f3 [#655] storage: Drop ErrorHandler
The only one usage was for logging.
Now logging performed by storage anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
137e987a4e [#655] storage: Drop LazyHandler
LazyHandler is implemented and used incorrectly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
4d5be5ccb5 [#811] ape: Update policy-engine module version and rebase
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-16 11:31:37 +03:00
a952a406a2 [#506] container: Use uint64 for epoch type
It is `uint64` in netmap source interfaces and other code.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
f04806ccd3 [#506] container: Use user.ID in DeletionInfo response
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
8088063195 [#787] netmap: Refactor NewEpoch method
Split for user and control methods.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:22:31 +03:00
c8a62ffedd [#787] morph: Calculate VUB and nonce when hash is nil
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
2393d13e4d [#787] morph: Return VUB for IR service calls
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
518f3baf41 [#787] morph: Return VUB from Invoke method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
bdfa523487 [#787] proto: Add VUB field for IR service
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
0f75e48138 [#796] policer: Fix tombstone objects replication
Tombstone objects must be replicated to all container nodes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 13:39:33 +03:00
7cdae4f660 [#792] proto: Regenerate with fixed version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:09:13 +00:00
9133b4389e [#788] objectsvc: Fix formatting (gofumpt)
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:27:32 +03:00
1b22801eed [#788] engine: Fix flaky tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:25:46 +03:00
3534d6d05b [#794] objectsvc: Return accidentally removed acl checks for Head
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 17:13:58 +03:00
66848d3288 [#770] cli: Add methods to work with APE rules via control svc
* Add methods to frostfs-cli
* Implement rpc in control service

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
8e11ef46b8 [#770] object: Introduce ape chain checker for object svc
* Introduce Request type converted from RequestInfo type
  to implement policy-engine's Request interface
* Implement basic ape checker to check if a request is
  permitted to be performed
* Make put handlers use APE checker instead EACL

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
5ec73fe8a0 [#770] node: Introduce ape chain source
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
3a2c319b87 [#770] control: Generate gRPC methods to manipulate APE chains
* Define new types and gRPC methods to manipulate APE chains
  in control service.
* Stub gRPC handlers for the generated methods.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
70ab1ebd54 [#763] metrics: Add container_objects_total metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
9c98fa6152 [#763] metabase: Add container objects counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
226e84d782 [#684] node: Add skipped objects count to evacuation result
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
523fb3ca51 [#684] proto: Add skipped count to evacuation status response
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
74c91eeef5 [#777] client: Refactor PrmContainerList, PrmObjectSearch usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-06 06:50:11 +00:00
20d6132f31 [#531] signSvc: Add SetMarshaledData method call
To reduce memory allocations add `SetMarshaledData` method call
to return already marshalled data in next `StableMarshal` calls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-02 17:34:33 +03:00
a26483fc30 [#749] morph: Fix panic when closing morph client
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-11-01 10:48:10 +00:00
c80b46fad3 [#754] blobstor: Estimate compressability
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 11:24:32 +03:00
05b508f79a [#772] proto: Fix file ending
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:04 +03:00
79088baa06 [#772] node: Apply gofumpt
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
00aa6d9749 [#633] shard/test: Fix TestCounters()
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
261d281154 [#762] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 11:22:29 +00:00
869518be0a [#728] writecache: Fix Badger writecache race.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-30 18:36:41 +03:00
d4b6ebe7e7 [#725] writecache: Fix metric values
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-27 12:22:29 +03:00
121f5c4dd8 [#757] ir: Do not exclude node in maintenance mode from netmap
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-26 10:50:32 +03:00
9f7c2d8810 [#752] innerring: Simplify keyPosition()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
cddc58ace2 [#752] innerring: Optimize keyPosition()
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
              │      old       │                 new                 │
              │     sec/op     │   sec/op     vs base                │
KeyPosition-8   2771.50n ± 10%   40.32n ± 4%  -98.55% (p=0.000 n=10)

              │     old      │                  new                  │
              │     B/op     │     B/op      vs base                 │
KeyPosition-8   1.531Ki ± 0%   0.000Ki ± 0%  -100.00% (p=0.000 n=10)

              │    old     │                new                 │
              │ allocs/op  │ allocs/op  vs base                 │
KeyPosition-8   21.00 ± 0%   0.00 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
0a9830564f [#752] morph: Adopt neo-go RPC statuses
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
6950312967 [#752] morph: Drop loop copy kludges
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
fe1acf9e9a [#750] morph: Remove deprecated channel use
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:57:50 +00:00
c0b86f2d93 [#642] writecache: Remove usage of close channel in badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:57:50 +00:00
b0cf100427 [#49] node: React on SIGHUP only when node in READY state
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:55:29 +00:00
58b6224dd8 [#747] client: Refactor PrmObjectPutInit usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-10-20 11:55:40 +00:00
12b7cf2533 [#747] client: Refactor PrmObjectPutSingle usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-10-20 11:55:40 +00:00
dc4d27201b [#733] morph: Fix delete container signature check
Committed invalid condition, it was just for debug.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-19 18:07:37 +03:00
189dbb01be [#733] frostfs-cli: Add control ir remove-container
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-19 16:22:18 +03:00
f2437f7ae9 [#734] shard: Fix Delete method
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
7e0c5a55de [#734] writecache: Fix flush
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:17 +03:00
3a997d1207 [#680] metrics: Initialize log metrics together with services
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
994f48f8bb [#680] metrics: Export log and morph with script
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
aca11d7474 [#735] policer: Allow to provide metrics from the outside
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-11 15:14:13 +03:00
5e229dc248 [#701] metrics: add metric to evaluate policer performance
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-10-09 19:02:08 +00:00
4caa934eea [#729] containersvc: Remove load announcement
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-09 19:01:13 +00:00
d07afd803c [#726] writecache: Fix small object flush for Badger
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:50 +03:00
997ac7cd8d [#726] writecache: Fix small object flush for BBolt
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:44 +03:00
c1e4130020 [#146] node: Add trace_id to logs
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
3889e829e6 [#667] writecache: Add logs for report error func in tests
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-09-14 17:02:54 +00:00
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-14 10:39:18 +03:00
8a81af5a3b [#653] Add context parameter to Open functions
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
a716db99db [#668] shard/test: Do not alter rootPath option
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
36759f8434 [#668] shard/test: Properly check event processing
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
39879fa868 [#668] shard/test: Add dontRelease options
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
268adb79cb [#668] shard/test: Simplify shard construction
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
429f941cda [#668] shard/test: Release shard in t.Cleanup()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
382eb8a485 [#668] shard/test: Disable GC where it is not needed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
42696016de [#668] shard: Close stopChannel in GC
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
bdecfbc1be [#668] shard/test: Move tests to the main package
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
da8f384324 [#668] shard/test: Fix typo in existence
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aeeb8193d2 [#676] node: Fix header source creation when checking eacl
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 17:06:54 +03:00
88d50e4c77 [#656] policer: Add "bad" testcase
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 08:04:59 +00:00
a54b4472de [#674] network: Close connections on address updates
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-04 15:34:24 +03:00
7456c8556a [#536] blobovnicza: Add blobovniczatree DB cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:53:11 +03:00
c672f59ab8 [#536] blobovnicza: Drop cache
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:51:26 +03:00
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-31 08:39:42 +00:00
4dff9555f1 [#568] writecache: Improve flushing scheme for badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:22:28 +00:00
806cc13d9f [#658] client: Refactor PrmObjectGet/Head/Range usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-30 17:13:23 +00:00
1daef2ceeb [#660] writecache: Fix remaining addr2key uses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-30 17:12:33 +00:00
fe5aa06a75 [#665] node: Bind length of copies number to number of replicas
Allow to use one digit in copies number array for backward compatibility.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:11:55 +00:00
55b82e744b [#529] objectcore: Use common sender classifier
Use common sender classifier for ACL service and format validator.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ae81d6660a [#529] objectcore: Fix object content validation
There are old objects where the owner of the object
may not match the one who issued the token.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ab2614ec2d [#528] objectcore: Validate token issuer
Add token issuer against object owner validation.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:10:10 +03:00
4ea0df77d0 [#574] policer: Check if the container was really removed
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
554ff2c06b [#574] core: Extend Source interface with DeletionInfo method
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
9072772a09 [#649] shard/test: Increase GC remover interval
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
c4db8e7690 [#637] shard/test: Fix data race
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
f8ba60aa0c [#648] objsvc/delete: Handle errors in Go style
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
d2084ece41 [#648] objsvc/delete: Remove redundant logs
We never propagate delete requests to the container node, because
tombstone broadcast is done via PUT. No need to pollute logs.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
40b556fc19 [#647] objsvc/search: Improve testing coverage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
4db2cbc927 [#647] objsvc/search: Wrap in uniqueIDWriter during parameter setting
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
966ad22abf [#647] objsvc/search: Simplify error handling
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
56f841b022 [#647] objsvc/search: Remove TraverserGenerator wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
ba58144de1 [#647] objsvc/search: Remove netmap.Source wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
c9e3c9956e [#643] objsvc/put: Unify extraBroadcastEnabled usage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
facd3b2c4b [#643] objsvc/put: Unify placement iterators
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
3fcf56f2fb [#643] objsvc/put: Copy config to distributedTarget
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
96e690883f [#638] Unify test loggers
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
322c1dc273 [#638] Use test.NewLogger() in tests
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"

-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
02b03d9c4f [#638] logger: Remove sampling from test loggers
Losing logs is always a bad idea, especially when we debug tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
82cc453be9 [#xx] shard: Fix data race in metrics tests
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 10:26:12 +00:00
345a1a69a2 [#635] Use internal key type when deleting from badger wc
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-22 10:53:19 +03:00
23be3eb627 [#574] tree: Check if container is really removed
* Use DeletionInfo method from morph client to check if
  the container has been really removed from neo-go

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-21 12:50:20 +03:00
42fb6fb372 [#574] morph: Add DeletionInfo method for morph client
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-21 12:49:06 +03:00
354a92ea2c [#602] blobovnicza: Add leaf width implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
4d9a6c07fb [#618] core: Replace fmt.Sprintf with strconv.FormatUint
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00
f2811f8585 [#602] metrics: Add blobovnicza items counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 13:01:27 +03:00
c4e1d8eb07 [#602] node: Fix blobovnicza typos
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 11:14:10 +03:00
10e63537b2 [#602] metrics: Rename blobovnicza size metric
`Size` is not size, but open db size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
809e97626b [#602] blobovnicza: Fix size counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
2e49d7ea7e [#602] blobovnicza: Init before using
Fix blobovnicza size: after restart size metric resets.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
f7042c5a6f [#609] Replace zaptest.NewLogger() with zap.L()
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"

-zaptest.NewLogger(t)
+zap.L()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:51:10 +00:00
127c676786 [#607] *: Use keys.PublicKeys.Copy() where possible
Semantic patch:
```
@@
var dst identifier
var src identifier
var keys identifier
@@
 import keys "github.com/nspcc-dev/neo-go/pkg/crypto/keys"

-dst := make(keys.PublicKeys, len(src))
-copy(dst, src)
+dst := src.Copy()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
e604a3d749 [#607] *: Use zap.Stringer() where possible
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
 f(
    ...,
-    zap.String(t, a.String()),
+    zap.Stringer(t, a),
    ...,
)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
a8de37c8a2 [#607] *: Remove redundant if on error returns
Semantic patch:
```
@@
@@
-if err != nil { return err }
-return nil
+return err
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
5a51b78946 [#620] object: Send status response for server-side streams
Previously status responses were wrapped in the gRPC error and thus
couldn't be correctly handled on client.

Introduced in c2617baf63, thanks @ale64bit for having found.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 12:52:38 +00:00
6407bb5bd1 [#619] node: Fix object put when copies numbers contains only zeros
In this case object should placement according to replicas.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-17 11:31:34 +03:00
5335e7089e [#615] pilorama: Speedup TestForest_ApplyRandom()
Some of our pilorama tests fail on CI.
The reasons are not obvious, but one possible improvement
is using `WithNoSync` option for these. It should have much effect,
because we are writing on the tmpfs, but doesn't hurt anyway.

If I replace `t.TempDir()` with a local directory, test execution time
goes down from 5s (sync) to 0.4s (nosync), which is the same time as
with `t.TempDir()`. Maybe we have some strange CI configuration.

```
panic: test timed out after 10m0s
running tests:
	TestForest_ApplyRandom (8m22s)
	TestForest_ApplyRandom/bbolt (8m21s)
...
goroutine 170 [syscall]:
syscall.Syscall(0xc000100000?, 0xc00047b758?, 0x6aff9a?, 0xc00041c1b0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:69 +0x27
syscall.Fdatasync(0x9e35c0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/zsyscall_linux_amd64.go:418 +0x2a
go.etcd.io/bbolt.fdatasync(0xc000189000?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 19:09:08 +00:00
2efe9cc1be [#585] writecache: Fix DB counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
58c8722c81 [#585] fstree: Add optional file counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
baad49990c [#585] fstree: Return logical error if object deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
0c52186572 [#585] fstree: Remove unused method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
eec97d177e [#585] writecache: Count items periodically
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
d15199c5d8 [#596] engine: Consider context errors as logical
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 10:39:41 +03:00
88b6755c5e [#598] Fix use-after-close bug in badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-15 07:04:06 +00:00
ae8be495c8 [#xx] Avoid manual management of files in tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 14:01:39 +03:00
376f03a445 [#598] Hold mode mutex when setting mode
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 07:12:27 +00:00
21800e9fcc [#162] core: Move literals to constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
b44a8dd46c [#597] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:55:51 +03:00
e084c47bd6 [#248] innerring: Remove audit from tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
779da6ec35 [#248] morph: Remove audit contract name
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
55ce4dc075 [#248] morph: Remove obsolete network parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
4ad0ebb32f [#565] Add metrics for current GRPC endpoint status
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-10 16:37:49 +00:00
c3e23a1448 [#578] gendoc: Allow to override flags
The command is used in multiple places across the whole FrostFS
ecosystem. While we want to have uniform interfaces everywhere,
sometimes we can't: already defined global flags can be harder to change
because of our obligations to the users. Cobra framework doesn't allow
conflicting flags (we can have global ones), so allow to override them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
6bcba27757 [#578] gendoc: Remove flag shorthands
Make it harder to encounter conflicts in already existing commands.
Because the command is executed once, I don't think the usability is
worse.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
dca31e8888 [#578] gendoc: Add date to man pages
```
HISTORY
       8-Aug-2023 Auto generated by spf13/cobra
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
b02a1a34c1 [#578] gendoc: Allow to customize man pages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
082370583f [#578] gendoc: Replace NSPCC with FrostFS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
8f994163ee [#586] Fix writecache benchmarks and refactor hacky NeedsCompression
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 08:05:18 +00:00
023b90342c [#584] Disable compression in badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 10:14:06 +03:00
d641cba2fc [#587] Do not use math/rand.Read
Fix staticcheck warnings after go1.20 update.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 10:26:53 +00:00
ae322e9f73 [#576] Set SyncWrites for badger writecache by default
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 11:28:26 +03:00
8d589314b5 [#560] node: Fix Put in multi REP with intersecting sets of nodes
Once the node was processed it skipped, at the step of forming
result in case when all nodes skipped, because processed for
previous REP, service mark the whole request as incomplete.

Example of policies which are unblocked:
- REP 1 REP 1 CBF 1
- REP 4 IN X REP 4 IN Y
  CBF 4
  SELECT 2 FROM FX AS X SELECT 2 FROM FY AS Y
  FILTER Country EQ Russia OR Country EQ Sweden OR Country EQ Finland AS FY
  FILTER Price GE 0 AS FX

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-08 10:22:53 +00:00
7da4306e38 [#575] writecache: Fix log level for badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-08 08:16:36 +00:00
0e697266c3 [#563] writecache: Fix metrics and bolt delete
Estimate cache size after delete objects to update metric.
Update counters on small object deletion.
Do not count bbolt DB file as FSTree object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-07 12:53:28 +00:00
1a0cb0f34a [#421] Try using badger for the write-cache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-07 08:16:57 +00:00
65c72f3e0b [#559] Remove manual path handling in fstree tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-03 10:03:41 +03:00
1e8b4b8a17 [#557] services: Regenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-02 13:37:40 +00:00
b3695411d9 [#553] eacl: Fix bug with casting to ObjectAccessDenied error
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-02 07:22:48 +00:00
35dc64bd7b [#547] metabase: Fix datarace in tests
Quite an old one bf9e938a3b.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
05ac9e3637 [#547] objectsvc: Work with traversal struct from a single thread
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
7b0fdf0202 [#533] services: Assume API supports status codes
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
ec8b4fdc48 [#541] writecache/test: Close writecache on exit
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-29 10:57:25 +00:00
ad5f527bd3 [#541] writecache/test: Remove initWC()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-29 10:57:25 +00:00
ea32913430 [#543] putsvc: Fix PutSingle implementation
Add Lock and Delete handlers to local PutSingle.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-28 12:09:41 +00:00
99bb488ebd [#539] getsvc: Write payload direct to out stream
To reduce memory allocations.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-27 17:02:08 +03:00
286242cad0 [#539] getsvc: Use buffer to assemble object
To reduce memory consumption.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-27 17:02:08 +03:00
5ff82ff04f [#6] services/object: Simplify local/remote targets
We do not use the return result from Close() and we always execute both
methods in succession. It makes sense to unite them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
448b48287c [#6] services/util: Do not panic in sign function
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
c2617baf63 [#6] services/util: Remove remaining stream wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:09 +03:00
372160d048 [#6] services/util: Remove SignService.HandleUnaryRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:36:08 +03:00
fef172c5b0 [#6] services/util: Simplify response.Service
It has only 1 parameter, which is obligatory.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:29:57 +03:00
eed594431f [#335] treesvc: Add GetSubTree ordering unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
af82c2865e [#335] treesvc: Fix inmemory unit tests and nil meta items
Bolt forest saves empty slice of items. Now inmemory forest
does it the same way.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
b4e72a2dfd [#335] treesvc: Sort nodes by Filename in GetSubTree
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
f0355a453e [#463] policer: Remove capacity rebalance logic
Current implementation has some quirks. For example,
using only half of object.put.pool_size_remote threads
tells replicator that is node is 50% loaded,
but in reality we could be putting lot's of big objects.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
8b78db74bc [#463] replicator: Add tracing span
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
8966dd8e35 [#463] putsvc: Use PutSingle RPC for remote target
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
3b66f98f27 [#519] Use Address.Equals in policer tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-17 10:45:07 +00:00
3e8de14e7d [#382] evacuate: Fix unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 10:24:05 +00:00
397131b0ea [#520] objectcore: Refactor format validator
Remove redundant FIXME.
Move error to consts.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 07:07:21 +00:00
11027945d8 [#479] writecache: Fix writecache fstree flush premature ctx cancel
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-14 10:25:52 +03:00
8a9fc2c372 [#510] treesvc: Rename tableFromBearer to useBearer
With impersonation, the old name is no longer descriptive.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
b8bcfac531 [#510] treesvc: Fix panic in bearer token processing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
24eb988897 [#294] deletesvc: Drop cast
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
c83e7c875f [#294] searchsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
e8091101c7 [#294] searchsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
ec9b738465 [#294] putsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
800a685e84 [#294] putsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
1420b8b9ea [#294] deletesvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
a476d8285a [#294] deletesvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
70a1081988 [#294] aclsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
18d8898b00 [#294] aclsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
61541eaec2 [#294] aclsvc: Refactor checker constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00