TrueCloudLab
/
frostfs-node
18
1
Fork 21
Code Issues 89 Pull Requests 8 Actions Releases Activity
4,676 Commits
7 Branches
23 Tags
48 MiB
Commit Graph

1110 Commits (d0f64c23a5d96937210c1b9aafbdd235674fab57)

Author SHA1 Message Date
Alejandro Lopez 26acf5689e [#92] Ensure policer objects cannot be worked on concurrently
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-03 07:05:31 +00:00
Alejandro Lopez 3223402c90 [#92] Embed policer's objectsInWork mutex 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-03 07:05:31 +00:00
Alejandro Lopez f9730f090d [#92] Refactor policer and add some unit tests 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-03 07:05:31 +00:00
Airat Arifullin f437ab8f15 [#197] object: Make Delete method return correct status
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-06-30 12:58:56 +00:00
Evgenii Stratonikov d0ab552a90 [#478] *: Fix funlen linter warnings
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-28 15:01:49 +00:00
Evgenii Stratonikov 8a4e250dae [#468] *: replace outdated TODO crypto-related links
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-28 12:13:20 +00:00
Evgenii Stratonikov 4f413fe86e [#1] treesvc: Properly check for secure transport 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-28 12:13:20 +00:00
Evgenii Stratonikov f4c71cea65 [#1] *: Replace outdated FIXME/TODO links 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-28 12:13:20 +00:00
Evgenii Stratonikov 0c866f62c5 [#473] placement: Fix backwards compatibility for `copies_number`
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-27 11:13:16 +03:00
Anton Nikiforov 71889234b7 [#449] tree: Allow reading requests signed by keys from allow list 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-06-23 14:44:19 +03:00
Evgenii Stratonikov 40eae22109 [#460] services/util: Remove `CreateRequestStreamer`
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-21 17:08:04 +03:00
Evgenii Stratonikov a64dc9ad70 [#460] services/util: Remove `HandleServerStreamRequest` 
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-21 17:08:01 +03:00
Evgenii Stratonikov 167a67f0b8 [#460] services/util: Remove `HandleUnaryRequest` 
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-21 17:07:56 +03:00
Evgenii Stratonikov 785d81a68a [#460] services/object: Reduce `distibutedTarget` memory footprint 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-06-21 17:07:50 +03:00
Dmitrii Stepanov 059e9e88a2 [#373] metabase: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Anton Nikiforov 50caa388b0 [#303] ir: Use pub key when validate container deletion
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-06-20 08:02:48 +00:00
Dmitrii Stepanov c348ae35b0 [#424] metrics: Drop embedded metrics 
It was not obvious where metrics are used.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-14 14:53:32 +03:00
Alejandro Lopez 4887f489a1 [#17] Add morph client metrics
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-06-13 14:06:04 +03:00
Airat Arifullin 90e9247b69 [#371] ir: Add morph cache metrics
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-06-13 10:10:13 +00:00
Dmitrii Stepanov 957a43a124 [#266] services/tree: Add sync check
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Do not accept requests until initial sync is finished.
`Apply` is deliberately left out -- we don't want to miss anything new.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-13 10:00:45 +00:00
Dmitrii Stepanov e69a1e8482 [#266] services/tree: Return operation log up to some height 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-13 10:00:45 +00:00
Dmitrii Stepanov 0400153b7d [#424] object: Do not store large slices in pool 
Dynamically growing an unbounded buffers can cause a large
amount of memory to be pinned and never be freed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-13 07:19:55 +00:00
Dmitrii Stepanov dcdfb6ed41 [#412] node: Use observability interceptors 
Use metrics and tracing interceptors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Dmitrii Stepanov 74578052f9 [#412] node: Replace tracing package 
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-01 13:23:11 +00:00
Alejandro Lopez faca861451 [#411] Remove unnecessary pointers for sync objects
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-31 10:19:14 +00:00
Alejandro Lopez bc34fee6a7 [#370] Add tree service metrics
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-26 13:39:12 +00:00
Dmitrii Stepanov 9119199f6e [#397] cli: Fix evacuation method names
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-26 11:28:30 +03:00
Dmitrii Stepanov f7c0b50d70 [#329] cli: Add async evacuate commands 
Add start, stop evacuate and evacuate status commands.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-19 08:43:52 +00:00
Dmitrii Stepanov e4889e06ba [#329] node: Make evacuate async 
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-19 08:43:52 +00:00
Dmitrii Stepanov 100b1b5128 [#329] node: Add async evacuate proto methods 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-19 08:43:52 +00:00
Alejandro Lopez 13a7a90101 [#355] Increase tree svc client cache size to test hypotheses
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-17 14:11:46 +03:00
Alejandro Lopez a6ee7a3087 [#324] Add replicator metrics
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-12 11:00:17 +00:00
Alejandro Lopez 90799497d3 [#114] Add remove-node IR control command
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-05-10 14:31:44 +00:00
Evgenii Stratonikov b2ffd7df53 [#291] object: Use PayloadSizeLimiter from SDK
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-05 19:07:06 +03:00
Evgenii Stratonikov 35ea207df6 [#291] object: Split validating target in two 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-05-05 19:07:06 +03:00
Pavel Karpy ee58b390bb [#221] node: Allow using vector `copies_number`
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Also, take into account that value in general (it was not used before at
all).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-05 16:07:13 +00:00
Pavel Karpy 479c5a65e1 [#322] node: Fix tree svc panic
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-05-04 19:48:51 +03:00
Dmitrii Stepanov 529d0bc710 [#302] tree: Drop unused ctx
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-02 16:54:06 +03:00
Dmitrii Stepanov eca5c210dd [#299] evacuate: Add context cancel checks
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-05-02 15:55:30 +03:00
Dmitrii Stepanov 22d47376a6 [#277] getsvc: Refactor errors
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Move errors to separate files.
Use zap.Error for error logging.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-28 14:03:12 +00:00
Dmitrii Stepanov 45438e7b06 [#277] getsvc: Rename and reorder code 
Rename execCtx to request.
Move code to appropriate files.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-28 14:03:12 +00:00
Dmitrii Stepanov 1440450606 [#277] getsvc: Drop cyclic struct dependency 
Drop cyclic dependency between execCtx and Service.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-28 14:03:12 +00:00
Dmitrii Stepanov 591c4e7d50 [#277] getsvc: Move headOnly to request params 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-28 14:03:12 +00:00
Dmitrii Stepanov 265d2326a0 [#277] getsvc: Extract remote storage 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-28 14:03:12 +00:00
Dmitrii Stepanov 30e1b62b67 [#277] getsvc: Fix service deps 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-28 14:03:12 +00:00
Dmitrii Stepanov 8fc082b688 [#277] getsvc: Do not return status error 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-28 14:03:12 +00:00
Evgenii Stratonikov f1ea8fec93 [#248] object_manager: Remove storage group 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 17:00:43 +03:00
Evgenii Stratonikov 7f49f07255 [#248] services: Remove audit service 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 17:00:43 +03:00
Evgenii Stratonikov f73ac6e02d [#290] control: Use generics for response wrappers
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-28 10:57:24 +03:00
Alejandro Lopez ff25521204 [#270] Add IR epoch tick control call
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-04-28 07:57:00 +00:00
Evgenii Stratonikov dfe4ada838 [#285] lint: Resolve revive/if-return 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:08 +03:00
Evgenii Stratonikov f07e2d4812 [#285] lint: Fix revive/unused-parameter 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-26 17:35:04 +03:00
Airat Arifullin 9d01029733 [#166] node: Parallelize background tree service sync by batching
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Merge operations

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-26 10:17:56 +00:00
Airat Arifullin 299b24b974 [#166] node: Parallelize background tree service sync by batching 
* Concurrently dispatch TreeApply operations for batching in forest

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-26 10:17:56 +00:00
Denis Kirillov dce5924a89 [#229] services/tree: Use bearer owner as signer 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 10:23:33 +03:00
Alexey Vanin 89530534a1 [#229] service/tree: Disable container owner check in tree service 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-04-26 10:23:33 +03:00
Alexey Vanin c04f6c5e59 [#229] acl: Allow Impersonate 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-04-26 10:23:33 +03:00
Dmitrii Stepanov 3d43b0f7f9 [#265] node: Fix after SDK & API-Go version up
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-18 12:09:19 +03:00
Dmitrii Stepanov f07d4158f5 [#249] node: Drop subnet from IR and morph 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-17 09:46:02 +03:00
Evgenii Stratonikov 8466894fdf [#250] control: remove `DumpShard` and `RestoreShard` RPC
ci/woodpecker/push/pre-commit Pipeline was successful Details 
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 12:28:49 +00:00
Pavel Karpy 560f73ab7e [#247] node, ir: Drop reputation related code 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-14 14:44:14 +03:00
Dmitrii Stepanov 6121b541b5 [#242] treesvc: Add tracing spans
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Dmitrii Stepanov d62c6e4ce6 [#242] node: Add tracing spans 
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Dmitrii Stepanov 200fc8b882 [#242] put: Pass context to relay function 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-14 10:25:53 +00:00
Evgenii Stratonikov cffcc7745e [#240] logs: Factor out common service log messages
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Evgenii Stratonikov 0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Evgenii Stratonikov 04727ce1d6 Revert "[#135] signature: Add tracing" 
This reverts commit 5778980252.
 2023-04-12 16:57:02 +03:00
Evgenii Stratonikov 08769f413f Revert "[#135] acl: Add tracing spans" 
This reverts commit b2ca730547.
 2023-04-12 16:54:13 +03:00
Dmitrii Stepanov 5778980252 [#135] signature: Add tracing
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Add tracing to verify request and sign response.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Dmitrii Stepanov b2ca730547 [#135] acl: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Dmitrii Stepanov 0920d848d0 [#135] get-object: Add tracing spans 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Dmitrii Stepanov 5af9f58469 [#135] tracing: Add tracing to node gRPC endpoints 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-12 06:52:00 +00:00
Dmitrii Stepanov 93eba19a8e [#223] objectsvc: Refactor split-tree traverse 
Resolve funlen & gocognit linters for traverseSplitChain method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 14:16:34 +03:00
Dmitrii Stepanov 2ed9fd3f94 [#223] objectsvc: Refactor request parameters 
Resolve containedctx linter for commonPrm.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 14:16:34 +03:00
Dmitrii Stepanov ccf8463e69 [#223] controlsvc: Drop unnecessary nolint 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-10 14:16:34 +03:00
Evgenii Stratonikov 0b9622c418 [#191] control: Add Doctor RPC 
Doctor RPC performs complex operations on the storage engine.
Currently only duplicate removal is supported.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-07 17:25:50 +00:00
Dmitrii Stepanov c236b54a65 [#212] reputationsvc: Resolve funlen linter
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for Calculator.iterateDaughter method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov 7ebbfa3358 [#212] reputationsvc: Resolve linters and rename 
Resolved containedctx linters.
Renamed context structs and interfaces to more understandble names.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov 469e8a6e59 [#212] reputationsvc: Resolve containedctx linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov e2f13d03d7 [#222] auditsvc: Refactor PoR audit
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for Context.checkStorageGroupPoR method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Dmitrii Stepanov e8d340287f [#222] auditsvc: Refactor audit task 
Resolve containedctx linter. Cancel task by listen cancel.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Dmitrii Stepanov 3dbff0a478 [#222] auditsvc: Resolve containedctx linter 
Resolve containedctx linter for commonCommunicatorPrm type.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Airat Arifullin 56282edf02 [#166] node: Parallelize background tree service sync
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Run sync task for nodes in parallel within errgroup worker pool

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 04:43:32 +00:00
Dmitrii Stepanov 4ec69cbbf8 [#203] control: Fix test 
Fix shard compare loop.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov 9e2df4b7c7 [#203] node: Fix double imports 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov c58ab0c369 [#193] getsvc: Reduce private key requests
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Get private key only once for request forwaring.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 89924071cd [#193] getsvc: Edit request forwarder signature 
Pass context to forwarder direct, without closure.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 6c7b708a98 [#193] getsvc: Refactor get range params creation 
Resolve funlen linter for toRangePrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov b0786d2e5c [#193] getsvc: Refactor get params creation 
Resolve funlen linter for toPrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov f889893216 [#193] getsvc: Refactor head param creation 
Resolve funlen linter for toHeadPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 91ead04fa4 [#193] getsvc: Resolve funlen linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 1bf21dbb47 [#193] getsvc: Resolve context linters 
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 206458c841 [#217] containersvc: Resolve containedctx linter
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:00 +00:00
Dmitrii Stepanov 279261ace3 [#217] containersvc: Refactor route passing 
Resolve containedctx for routeCtx.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:00 +00:00
Dmitrii Stepanov d6486d172e [#210] policier: Refactor nodes processing
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for processNodes method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov 080be5cfcd [#210] policier: Refactor object placement 
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov 23575e1ac0 [#210] policier: Resolve contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov d85703a963 [#208] searchsvc: Refactor request forwarding
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen & gocognit linters for toPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 16:24:55 +03:00
Dmitrii Stepanov 0b38419fbf [#208] searchsvc: Resolve context linters 
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 16:24:50 +03:00
Dmitrii Stepanov 5f2a1531fe [#208] deletesvc: Resolve containedctx linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 16:24:40 +03:00
Dmitrii Stepanov 4941926c9d [#207] aclsvc: Drop outdated tag
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 13:22:14 +00:00
Dmitrii Stepanov 585415fa92 [#207] aclsvc: Refactor send checker 
Resolve funlen linter for putStreamBasicChecker.Send method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 13:22:14 +00:00
Dmitrii Stepanov 9ef790f782 [#207] aclsvc: Refactor object headers read 
Resolve funlen linter for readObjectHeaders method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 13:22:13 +00:00
Dmitrii Stepanov cd33a57f44 [#207] aclsvc: Refactor EACL check 
Resolve funlen linter for CheckEACL method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 13:22:13 +00:00
Dmitrii Stepanov ece6c820e7 [#199] putsvc: Refactor streamer initialization
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for initTarget method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-03 15:58:11 +00:00
Dmitrii Stepanov 27bdddc48f [#199] putsvc: Refactor put object 
Resolve containedctx linter for streamer and remote target

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-03 15:58:11 +00:00
Dmitrii Stepanov cecea8053a [#199] putsvc: Refactor streamer pool 
Resolve staticcheck linter for putBytesPool

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-03 15:58:11 +00:00
Dmitrii Stepanov 14d894178e [#199] putsvc: Refactor placement iterator 
Resolve funlen linter for iteratePlacement method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-03 15:58:11 +00:00
Dmitrii Stepanov 56161d39b4 [#168] node: Refactor container services 
Resolve containedctx for remoteLoadAnnounceWriter struct

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-31 09:32:59 +03:00
Airat Arifullin 221203beeb [#180] node: Refactor panics in unit test
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Replace panics in unit tests by require.NoError and t.Fatalf

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-03-29 12:39:07 +03:00
Denis Kirillov 342e571d89 [#159] Add handle __SYSTEM__ sys attributes
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-22 17:35:20 +03:00
Dmitrii Stepanov 49234b915e [#155] search-service: Fix search with ST
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Search should return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-22 07:21:09 +00:00
Dmitrii Stepanov 1637a3edce [#155] search-service: Add search with ST test 
In case of session token (ST) with object IDs search should
return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-22 07:21:09 +00:00
Evgenii Stratonikov cbc2efb1d6 [#156] object/get: Make toHeadPrm() pass gocognit
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov 3f6b962349 [#156] services/tree: Pass context to replicationWorker() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov 5368c4207a [#156] services/tree: Split syncLoop() in functions 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov 47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply() 
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Dmitrii Stepanov 44b86bac5a [#148] linter: Add contextcheck linter
ci/woodpecker/push/full-pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov 481a1ca6f3 [#148] linter: Add gocognit linter 
Code with high cognitive complexity is hard intuitively to understand

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov 97c36ed3ec [#148] linter: Add funlen linter 
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov cc8ff015b4 [#148] linter: Add containedctx linter 
Context has to be passed as an argument: https://pkg.go.dev/context

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:52:39 +03:00
Pavel Karpy abd21f8099 [#136] Revert "[#2260] services/object: Do not assemble object with TTL=1" 
This reverts commit 2567f8020e. It assumes
that assembling logic could break some failover scenarios if request
forwarding is done. However, it also breaks requesting big objects via a
non-container node with TTL=2. Failover has been rechecked without that
commit and no problems were found. Any (if found) other bugs related to
the forwarding and object assembling must be solved more carefully.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-20 10:52:18 +03:00
Pavel Karpy 64bde68fb9 [#67] node: Accept expired locked objects 
Allow replication of any (expired too) locked object. Information about
object locking is considered to be presented on the _container nodes_.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-16 16:22:19 +03:00
Evgenii Stratonikov 22be532cbd object/put: Persist session token till the end of a session 
Previously a token could've expired in the middle of an object.PUT
stream, leading to upload being interrupted. This is bad, because user
doesn't always now what is the right values for the session token
lifetime. More than that, setting it to a very high value will
eventually blow up the session token database.

In this commit we read the session token once and reuse it for the whole
stream duration.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-16 06:45:50 +00:00
Dmitrii Stepanov ac0a278a05 [#85] get-service: Drop unused assemble flag 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Dmitrii Stepanov b8e93d4c08 [#85] get-service: Use assembler to assemble LOB 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Dmitrii Stepanov 07de839f18 [#85] get-service: Fix corrupted chain logic 
Should return an error in case of a broken LOB reference chain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Dmitrii Stepanov 2886b1581b [#85] get-service: Add unit tests 
Add unit tests to cover all assemble statements

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Dmitrii Stepanov 8b9e40a848 [#85] get-service: Add assembler 
Extract assemble logic to assembler

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Evgenii Stratonikov 6cd806f998 [#82] services/tree: Save last synchronized height in a persistent storage 
Remember the last synchronized height and use it after service restart.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 11:25:44 +00:00
Evgenii Stratonikov 6226c3ba86 [#129] policer: Use safer defaults 
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-10 10:59:15 +00:00
Evgenii Stratonikov f2250a316f [#129] tree: Do not remove tree if the netmap is empty 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-10 10:59:15 +00:00
Alexey Vanin 20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Dmitrii Stepanov 6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods 
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-02-28 13:36:25 +03:00
Alejandro Lopez 73bb590cb1 [#64] node: Use pool_size_local and separate pool for local puts 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-02-22 13:43:19 +03:00
Alejandro Lopez cb5468abb8 [#66] node: Replace interface{} with any 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-02-21 16:47:07 +03:00
Anton Nikiforov 22f3c7d080 [#1868] Reload config for pprof and metrics on SIGHUP 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov 2567f8020e [#2260] services/object: Do not assemble object with TTL=1 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Pavel Karpy 901d62567d [#57] node: Broadcast link objects 
It boosts object assembling by an _average_ container node.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-17 11:58:27 +03:00
Stanislav Bogatyrev cb016d53a6 [#1] Fix comments and error messages 
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
 2023-02-06 17:41:14 +03:00
Stanislav Bogatyrev c761a95eef [#1] Fix project name in control service 
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
 2023-02-06 17:41:14 +03:00
Evgenii Stratonikov d65a95a2c6 [#28] pilorama: Remove `LogMove` struct 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Evgenii Stratonikov 46c62be7e8 [#28] Fix linter issues 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Evgenii Stratonikov 67c97c6804 [#2210] services/tree: Drop messages not in queue 
Currently, under high load clients are blocked on channel send
and the number of goroutines can increase indefinitely.
In this commit we drop replication messages if send/recv queue is full
and rely on a background synchronization.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Evgenii Stratonikov 0d8366f475 [#2207] object/acl: Return status error for expired session token 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Evgenii Stratonikov 6efa93be0a [#1621] services/tree: Return `Apply` result asyncronously 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Evgenii Stratonikov 554b85411f [#2190] services/object: Log service error with INFO level 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-25 15:31:47 +03:00
Evgenii Stratonikov 3d1d2ee7b1 [#11] Regenerate proto files 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-01-12 08:55:47 +03:00
Evgenii Stratonikov f0be0befc5 [#5] services/object_manager: Use generic LRU cache 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-31 23:04:06 +03:00
Evgenii Stratonikov 1b3374ac7f [#5] services/tree: User generic LRU cache 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-31 23:04:06 +03:00
Evgenii Stratonikov 8f61cc1dcc [#5] policer: Use generic LRU client 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-31 23:04:06 +03:00
Evgenii Stratonikov 6f5edac730 [#2164] network/cache: Do not reconnect to failed clients immediately 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov b4e90cdf51 [#2165] pilorama: Optimize `TreeApply` when used for synchronization 
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov f9fcd85363 [#2165] services/tree: Remember starting height for the synchronization 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov 06137dbf8e [#2165] services/tree: Do not export `synchronizeAllTrees` 
It is used only privately.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov c299b98afe [#2165] services/tree: Parallelize synchronization 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov de9957e076 [#2165] services/tree: Always synchronize all containers 
In case of split-brain we must synchronize everything.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy 6a4e5e6f0a [#2144] node: Try node's private key if dynamic token fetching failed 
`GETRANGEHASH` request spawns `GETRANGE` requests if an object could not be
found locally. If the original request contains session, it can be static
and, therefore, fetching session key can not be performed successfully.
As the best effort a node could request object's range with its own key.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy 86a4fba571 [#2144] node: Clarify `KeyStorage.GetKey` method 
Actualize the doc, fix API status error return.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov 04b5ec759b [#2139] object/put: Use `sync.Pool` for temporary payloads 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov 9e0decd12d [#2162] services/tree: Close connection after the syncronization 
There was a goroutine leak here.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy 306609030a [#2159] node: Add tree replication timeout configuration 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov 3bb5a320d7 [#2154] services/tree: Do not log an error when synchronizing container of 1 node 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov 387d1e2977 [#2127] services/tree: Randomize node order for synchronization 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Evgenii Stratonikov b207dc424f [#2158] policer: Reduce default cache size 
We use cache to avoid policing the same object multiple times in a short
time span (< 30 seconds). If we have 200_000 objects in a blobstor, it is a bit useless
-- if it takes 1 second to process an object and we have `replicator.pool_size: 20`
in config, the next iteration will happen in 10_000 second which is much
larger than 30 second. However we still consume a lot of memory, so it
makes sense to use saner default.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy b413094704 [#2095] node: Fix collecting child objects 
Stop child objects collection if the last returned object (the most "left"
object in the collected chain) starts exactly from the `GETRANGE`'s `from`
value.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-12-30 11:07:35 +03:00
Pavel Karpy 350eecfa13 [#2095] node: Do not allow `GETRANGE` requests with zero length 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-12-30 11:07:35 +03:00
Pavel Karpy 923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Pavel Karpy d54022eacc [#2047] node: Do not send chunk twice on request forwarding 
That could happen if a node forwards request to a node that closed the
connection during the original object stream.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-12-02 11:27:48 +03:00
Evgenii Stratonikov bd25db5d4a [#1984] metrics: Use separate metrics for success/failed requests 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-12-01 14:18:10 +03:00
Evgenii Stratonikov e21c472dc7 [#1984] services/object: Increase put_req_count after the request is processed 
As it is specified in metrics description.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-12-01 14:18:10 +03:00
Pavel Karpy 51963abce7 [#1972] node: Fix errors comments in the Put service 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-30 16:58:52 +03:00
Evgenii Stratonikov 660c38d07e [#2062] services/policer: Use a proper key for object cache 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov 1779664644 [#2058] services/policer: Fix panic in `shardPolicyWorker` 
```
2022/11/15 08:40:56 worker exits from a panic: runtime error: index out of range [0] with length 0
2022/11/15 08:40:56 worker exits from panic: goroutine 1188 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0x1042b60, 0xc0015ae018})
	runtime/panic.go:1038 +0x215
github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
	github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x366
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x68
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Anton Nikiforov 9a20498f34 [#1940] Removing all trees by container ID if tree ID is empty in `pilorama.Forest.TreeDrop` 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2022-11-19 11:01:04 +03:00
Pavel Karpy 634792077e [#1502] node: Store lock object on every container node 
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Pavel Karpy d5a14041e0 [#2040] node: Do not attach tokens in the assembly process 
A container node is expected to have full "get" access to assemble the
object.
A non-container node is expected to forward any request to a container node.
Any token is expected to be issued for an original request sender not for a
node so any new request is invalid by design with that token.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Pavel Karpy fd61bdadcb [#2040] node: Attach original meta to the spawned requests 
Do not lose meta information of the original requests: cache session and
bearer tokens of the original request b/w a new generated ones. Middle
request wrappers should not contain any meta information, since it is
useless (e.g. ACL service checks only the original tokens).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Pavel Karpy 481b48b942 [#2028] node: Check session token's NBF and IAT 
ACL service did not check "Not Valid Before" and "Issued At" claims.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Pavel Karpy aadd2ad050 [#2028] node: Do not wrap malformed request errors 
After presenting request statuses on the API level, all the errors are
unwrapped before sending to the caller side. It led to a losing invalid
request's context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov 2522d924b9 [#2037] services/object: Fix concurrent map writes in traverser 
```
fatal error: concurrent map writes

goroutine 4337 [running]:
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*traversal).submitProcessed(...)
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:78
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*distributedTarget).iteratePlacement.func1()
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:198 +0x265
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov aa478f1def [#2024] services/object: Unify status errors 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov 3875fef542 [#2024] services/object: Cover corner cases for children OutOfRange 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Pavel Karpy aab398f4f5 [#1972] node: Do not save objects if node not in a container 
Do not use node's local storage if it is clear that an object will be
removed anyway as a redundant. It requires moving the changing local storage
logic from the validation step to the local target implementation.
It allows performing any relations checks (e.g. object locking) only if a
node is considered as a valid container member and is expected to store
(stored previously) all the helper objects (e.g. `LOCK`, `TOMBSTONE`, etc).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov a455ec18c3 [#2007] services/object: Allocate memory on-demand in GET_RANGE 
For big objects we want to get OutOfRange error before all the memory is
allocated.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov ff5526038d [#2007] services/object: Fix comment 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov 846ff515e6 [#1812] policer: Do not remove copies if there are maintenance nodes 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-26 19:13:17 +03:00
Leonard Lyubich 7b418c36b4 [#1930] services/session: Log calling `Create` RPC 
There is a need to check if session is opened during system
testing/debug.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-24 17:45:22 +03:00
Leonard Lyubich 60e9de8d63 [#1916] control: Check maintenance allowance on Control server 
In previous implementation turning to maintenance mode using NeoFS CLI
required NeoFS API endpoint. This was not convenient from the user
perspective. It's worth to move networks settings' check to the server
side.

Add `force_maintenance` field to `SetNetmapStatusRequest.Body` message
of Control API. Add `force` flag to `neofs-cli control set-status`
command which sets corresponding field in the requests body if status is
`maintenance`. Force flag is ignored for any other status.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-24 09:20:24 +04:00
Pavel Karpy e1be0180f6 [#1329] tree: Sync trees when a node first time appears in a container 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-21 18:47:56 +03:00
Pavel Karpy 1766ca2039 [#1902] tree: Allow synchronize all the container trees 
Add `SynchronizeAllTrees` method of the Tree service. It allows fetching
tree IDs and sync all of them. Share common logic b/w the new method and
the `SynchronizeTree`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Pavel Karpy 6d4beea187 [#1902] tree: Extend grpc service with `ListTrees` method 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Pavel Karpy 3aa9938b8f [#1902] Update protoc to `v3.21.7` 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Evgenii Stratonikov 9ec01bb9c1 [#1931] control: Allow to clear errors in `SetShardMode` RPC 
It hasn't been working since the initial implementation 7fb15fa1d0.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-20 15:51:31 +03:00
Evgenii Stratonikov 0d65888005 [#1910] .golangci.yml: Add `predeclared` linker 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-18 15:08:26 +03:00
Evgenii Stratonikov 1cb892c579 [#1910] .golangci.yml: Add `misspell` linker 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-18 15:08:26 +03:00
Evgenii Stratonikov d772e35aba [#1910] .golangci.yml: Add `godot` linker 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-18 15:08:26 +03:00
Pavel Karpy bc67f77d86 [#1329] tree: Fix sync applying 
Add the node position in a container and the container size to the CID
descriptor that is passed to the `TreeApply`. Previously, `checkValid` does
not allow any log operarations.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-17 20:53:34 +03:00
Pavel Karpy f5f560d903 [#1329] tree: Do not sync trees that are from external containers 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-17 20:53:34 +03:00
Leonard Lyubich 1406d096a2 [#1680] service/object: Fail all operations in maintenance mode 
Storage node should not provide NeoFS Object API service when it is
under maintenance.

Declare `Common` service that unifies behavior of all object operations.
The implementation pre-checks if node is under maintenance and returns
`apistatus.NodeUnderMaintenance` if so. Use `Common` service as a first
logical processor in object service pipeline.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-14 13:54:32 +04:00
Leonard Lyubich 05420173cc [#1894] services/object: Ignore unrelated sessions in client 
In some scenarios original session can be unrelated to the objects which
are read internally by the node. For example, node requests child
objects when removing the parent one.

Tune internal NeoFS API client used by node's Object API server to
ignore unrelated sessions in `GetObject` / `HeadObject` / `PayloadRange`
ops.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-14 12:37:31 +03:00
Pavel Karpy 13c4a9f4b8 [#1332] tree: Make `SignMessage` public 
It will allow reusing signing routine in other components
(e.g. `neofs-cli`).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-13 20:01:48 +03:00
Anton Nikiforov 8714fc42b5 [#1765] Use hex format to print storage node ID 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2022-10-13 12:55:21 +03:00
Pavel Karpy f037022a7a [#1770] logger: Refactor `Logger` component 
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-12 18:11:05 +03:00
Evgenii Stratonikov 4baf00aa21 [#1884] services/object: Fallback to GET in GET_RANGE 
Current spec allows denying GET_RANGE requests from other storage nodes.
However, GET should always be allowed and it is enough to perform
GET_RANGE locally

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-12 17:05:51 +03:00
Leonard Lyubich dde4d4df2a [#1878] services/object: Fix child check in GET 
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-12 16:56:37 +03:00
Evgenii Stratonikov 49eab6318c [#1867] control: Fix degraded-read-only mode parsing 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-12 11:20:48 +03:00
Evgenii Stratonikov c0199dee93 [#1867] services/control: Interpret empty list of IDs as all shards 
In neofs-cli the flag is still required, but `all` can be used to
process all shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-12 11:20:48 +03:00
Evgenii Stratonikov 19c0a74e94 [#1867] services/control: Allow to provide multiple shard IDs to some commands 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-12 11:20:48 +03:00
Leonard Lyubich 050ad2762c [#1680] replicator: Consider `NODE_UNDER_MAINTENANCE` as OK 
Node response with `NODE_UNDER_MAINTENANCE` status signals that the node
was switched to maintenance mode. There is a delay between the actual
switch and the reflection in the network map of up to one epoch. To
speed up the reaction to the maintenance, it is required to recognize
such node responses in the Policer.

Make `Policer.processNodes` to exclude elements with shortage decreasing
on `NODE_UNDER_MAINTENANCE` status response.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-11 12:54:27 +03:00
Leonard Lyubich e99e25b52f [#1680] replicator: Consider nodes under maintenance as OK 
Nodes under maintenance SHOULD not respond to object requests. Based on
this, storage node's Policer SHOULD consider such nodes as problem ones.
However, to prevent spam with the new replicas, on the contrary, Policer
should consider them normal.

Make `Policer.processNodes` to exclude elements if `IsMaintenance()`
with shortage decreasing.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-11 12:54:27 +03:00
Leonard Lyubich df5d7bf729 [#1680] replicator: Work with `netmap.NodeInfo` in `TaskResult` 
Make `replicator.TaskResult` to accept `netmap.NodeInfo` type instead of
uint64 in order to clarify the meaning and prevent passing the random
numbers.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-11 12:54:27 +03:00
Leonard Lyubich e6f8904040 [#1680] policer: Refactor tracking the processed nodes 
Add clear methods with docs. Use the methods instead of direct map
and bool instructions.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-11 12:54:27 +03:00
Anton Nikiforov feaa9eace7 [#1768] Add healthcheck method to the Tree service 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2022-10-11 12:53:45 +03:00
Leonard Lyubich 807c0a1321 [#1859] services/object: Do not session check relation in PUT 
It doesn't make sense to check object relation in session check of
`ObjectService.Put` RPC which has been spawned by `ObjectService.Delete`
with session. Session issuer can't predict identifier of the tombstone
object to be created.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-10 20:09:47 +03:00
Evgenii Stratonikov 4b005d3178 [#1840] blobstor: Return info about all components 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-10 11:14:55 +03:00
Evgenii Stratonikov 7356ee91ff [#1837] services/object: Optimize `uniqueIDWriter` 
Avoid encoding object ID to string.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-07 13:29:15 +03:00
Evgenii Stratonikov 2ac42b70ce [#1826] services/object: Parallelize object search 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-07 13:29:15 +03:00
Evgenii Stratonikov 0cb8e7f6f1 [#1837] services/object: Fix log messages 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-07 13:29:15 +03:00
Leonard Lyubich e54b52ec03 [#1420] object/acl: Fix correlation of object session to request 
In previous implementation of `neofs-node` app object session was not
checked for substitution of the object related to it. Also, for access
checks, the session object was substituted instead of the one from the
request. This, on the one hand, made it possible to inherit the session
from the parent object for authorization for certain actions. On the
other hand, it covered the mentioned object substitution, which is a
critical vulnerability.

Next changes are applied to processing of all Object service requests:
 - check if object session relates to the requested object
 - use requested object in access checks.

Disclosed problem of object context inheritance will be solved within

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-07 10:34:38 +03:00
Evgenii Stratonikov 5834f9807e [#1847] services/policer: Provide container ID in logs 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-07 09:58:16 +03:00
Leonard Lyubich 713aea06fa [#1681] node: Block only Object service ops under maintenance 
In previous implementation node blocked any operation of local object
storage in maintenance mode. There is a need to perform some storage
operations like data evacuation or restoration.

Do not call block storage engine in maintenance mode. Make all Object
service operations to return `apistatus.NodeUnderMaintenance` error from
each local op.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
 2022-10-05 11:41:49 +03:00
Evgenii Stratonikov 236414df49 [#1817] network: Allow to use network addresses from the iterator 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-04 12:12:18 +03:00
Evgenii Stratonikov 1edc048870 [#1697] services/object: Return proper error if session token is missing 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-30 16:13:01 +03:00
Evgenii Stratonikov 51b8f26a31 [#1806] services/control: Allow to flush write-cache 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-28 09:28:01 +03:00
Leonard Lyubich 93742d37b7 [#1793] node/netmap: Change interface of the latest network map reader 
Replace `ProcessCurrentNetMap` method of `NodeState` interface with
`ReadCurrentNetMap` one with two changes:
 * Replace network map type from NeoFS SDK package with the
   protocol-generated message. This replaces all the business logic to
   the application layer.
 * Support error return. This allows to cover problem node states.

Return an error from `NodeState.ReadCurrentNetMap` method implemeted
through `atomic.Value` if `Store` method has not been called yet.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-09-27 09:03:17 +03:00
Leonard Lyubich 485a5418d2 [#1793] node: Serve `NetmapService.NetmapSnapshot` RPC 
There is no more need to serve the same request on Control API.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-09-27 09:03:17 +03:00
Evgenii Stratonikov 898689ec14 [#1731] services/replicator: Unify Task interface with other parameters 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-24 13:47:48 +03:00
Evgenii Stratonikov 4e043a801c [#1731] services/control: Replicate object over network in EvacuateShard RPC 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-24 13:47:48 +03:00
Evgenii Stratonikov 091d7d30f6 [#1731] services/control: Allow to evacuate data from shard 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-19 11:33:52 +03:00
Evgenii Stratonikov b064fb24d8 [#1616] engine: Do not use batches in delete 
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-15 10:28:48 +03:00
Evgenii Stratonikov cda8f9df2e [#1786] services/control: Remove `WithDeletedObjectHandler` option 
Use storage engine directly instead. It is already provided in the
options.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-15 10:28:48 +03:00
Leonard Lyubich ae655b74f0 *: Exec `make fmts` 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-09-14 17:52:49 +04:00
Pavel Karpy 4083a62679 [#1628] tree: Pre-allocate errors in ACL checks 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy 5adf089c1d [#1628] tree: Log unacceptable bearer attachment 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy 2ffcd02ac3 [#1628] tree: Document ACL checks in tree service 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy 8d0906c6ab [#1628] tree: Skip eACL filters for tree requests 
Do not call `CalculateAction` for the eACL checks since it requires object
headers that are meaningless in the tree context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy 876e014b5d [#1628] tree: Make ACL checks the same way as for object requests 
1. Do not require a request to be signed by the container owner if a
bearer token is missing
2. Do not check the system role since public requests are not expected to
be signed by IR or a container node (unlike the object requests)

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy 4f18893d9b [#1628] node: Move common `EACLSource` interface to `core` pkg 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 10:33:50 +03:00
Pavel Karpy 4afb928ab6 [#1674] *: Expire entities after the expiration epoch 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-12 19:56:59 +04:00
Evgenii Stratonikov 9da5d784cb [#1630] neofs-node: Remove trees on container removal event 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-12 09:54:15 +03:00
Evgenii Stratonikov 0140ac354b [#1759] services/tree: Make logs more descriptive 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 12:57:33 +04:00
Evgenii Stratonikov 74c861342e [#1753] services/tree: Add parent ID to the `GetNodeByPath` response 
Currently, you need to use `GetSubTree` to get parent, which seems an
overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 10:52:37 +03:00
Evgenii Stratonikov 8a47f9ba12 [#1753] services/tree: Remove MaxGetSubTreeCount constant 
It was used only in `GetOpLog` and ignored on the server side.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 10:52:37 +03:00
Evgenii Stratonikov 9840936a4f [#1753] services/tree: Do not restrict depth in `GetSubTree` 
Previously, the depth was restricted because with BFS the amount of
nodes we have in memory blows up exponentially. With DFS is is linear,
so we can process trees of arbitrary depth.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-07 10:52:37 +03:00
Evgenii Stratonikov 180e5e938f [#1735] go.mod: Update API and SDK 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-08-30 12:27:46 +03:00
Pavel Karpy 0720d96c9d [#1687] go.mod: Update neofs-sdk-go 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-22 18:59:57 +03:00
Pavel Karpy 30341f2192 [#1687] *: Perform `go fmt` using go `v1.19` 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-22 18:59:57 +03:00
Pavel Karpy c2918fce3a [#1645] node: Support `EACL_NOT_FOUND` status 
Remove internal `ErrEACLNotFound` error.
Also, update `neofs-api-go` and `neofs-sdk-go` libraries.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-01 20:45:36 +03:00
Evgenii Stratonikov 4558f30575 [#1644] services/tree: Do not use deprecated gRPC options 
Get rid of `grpc.WithInsecure`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-07-30 18:39:08 +03:00