TrueCloudLab
/
frostfs-node
17
1
Fork 21
Code Issues 86 Pull Requests 6 Actions Projects Releases Activity
3,795 Commits
7 Branches
24 Tags
49 MiB
Commit Graph

2499 Commits (2f6757c828acf9235577175a4cbc8aa08fb36e98)

Author SHA1 Message Date
Dmitrii Stepanov c236b54a65 [#212] reputationsvc: Resolve funlen linter
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for Calculator.iterateDaughter method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov 7ebbfa3358 [#212] reputationsvc: Resolve linters and rename 
Resolved containedctx linters.
Renamed context structs and interfaces to more understandble names.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov 469e8a6e59 [#212] reputationsvc: Resolve containedctx linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:35:57 +00:00
Dmitrii Stepanov e2f13d03d7 [#222] auditsvc: Refactor PoR audit
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for Context.checkStorageGroupPoR method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Dmitrii Stepanov e8d340287f [#222] auditsvc: Refactor audit task 
Resolve containedctx linter. Cancel task by listen cancel.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Dmitrii Stepanov 3dbff0a478 [#222] auditsvc: Resolve containedctx linter 
Resolve containedctx linter for commonCommunicatorPrm type.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 17:29:13 +03:00
Dmitrii Stepanov fe87735073 [#219] morph: Refactor notary preparator
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for Preparator.Prepare method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:43:20 +03:00
Dmitrii Stepanov d07e40d6fe [#219] morph: Refactor moprh event listener 
Resolve funlen and gocognit linters for listener.listenLoop method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:43:20 +03:00
Dmitrii Stepanov 775179f823 [#219] morph: Refactor notary invoke 
Resolve funlen linter for Client.notaryInvoke method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:43:20 +03:00
Dmitrii Stepanov e815b19101 [#219] morph: Resolve containedctx linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-07 15:43:20 +03:00
Airat Arifullin 56282edf02 [#166] node: Parallelize background tree service sync
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Run sync task for nodes in parallel within errgroup worker pool

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-07 04:43:32 +00:00
Dmitrii Stepanov 4ec69cbbf8 [#203] control: Fix test 
Fix shard compare loop.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov f32f61df87 [#203] pilorama: Refactor tests 
Do not pass 0 as channel capacity.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov 8908798f59 [#203] node: Resolve unused vars 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov bab11492ad [#203] node: Resolve never used errors 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov 68a2f36636 [#203] morph: Mark depracated methods 
Skip staticcheck for depracated methods. Will be fixed soon.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Dmitrii Stepanov 9e2df4b7c7 [#203] node: Fix double imports 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-06 16:33:36 +03:00
Airat Arifullin ab891517de [#116] node: Fix bug with extra generated files in TestReload
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Create testNewEngine in engineWithShards without default opts

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-05 18:49:34 +03:00
Dmitrii Stepanov c58ab0c369 [#193] getsvc: Reduce private key requests
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Get private key only once for request forwaring.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 89924071cd [#193] getsvc: Edit request forwarder signature 
Pass context to forwarder direct, without closure.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 6c7b708a98 [#193] getsvc: Refactor get range params creation 
Resolve funlen linter for toRangePrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov b0786d2e5c [#193] getsvc: Refactor get params creation 
Resolve funlen linter for toPrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov f889893216 [#193] getsvc: Refactor head param creation 
Resolve funlen linter for toHeadPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 91ead04fa4 [#193] getsvc: Resolve funlen linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 1bf21dbb47 [#193] getsvc: Resolve context linters 
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:48 +00:00
Dmitrii Stepanov 206458c841 [#217] containersvc: Resolve containedctx linter
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:00 +00:00
Dmitrii Stepanov 279261ace3 [#217] containersvc: Refactor route passing 
Resolve containedctx for routeCtx.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:38:00 +00:00
Airat Arifullin 6f7b6a8813 [#116] node: Improve shard/engine construction in tests
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Introduce testEngineWrapper that can be constructed with different options

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
 2023-04-05 14:36:40 +00:00
Dmitrii Stepanov d6486d172e [#210] policier: Refactor nodes processing
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for processNodes method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov 080be5cfcd [#210] policier: Refactor object placement 
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov 23575e1ac0 [#210] policier: Resolve contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Evgenii Stratonikov 9098d0eec0 [#211] engine: Unify shard mode checks for tree operations
ci/woodpecker/push/pre-commit Pipeline was successful Details 
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-05 11:10:39 +00:00
Evgenii Stratonikov 760af6b912 [#211] fstree: Consider ENOSPC a logical error 
We already do this for file writing, however directory creation can also
fail.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-05 11:10:39 +00:00
Dmitrii Stepanov d85703a963 [#208] searchsvc: Refactor request forwarding
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen & gocognit linters for toPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 16:24:55 +03:00
Dmitrii Stepanov 0b38419fbf [#208] searchsvc: Resolve context linters 
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 16:24:50 +03:00
Dmitrii Stepanov 5f2a1531fe [#208] deletesvc: Resolve containedctx linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 16:24:40 +03:00
Dmitrii Stepanov 4941926c9d [#207] aclsvc: Drop outdated tag
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 13:22:14 +00:00
Dmitrii Stepanov 585415fa92 [#207] aclsvc: Refactor send checker 
Resolve funlen linter for putStreamBasicChecker.Send method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 13:22:14 +00:00
Dmitrii Stepanov 9ef790f782 [#207] aclsvc: Refactor object headers read 
Resolve funlen linter for readObjectHeaders method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 13:22:13 +00:00
Dmitrii Stepanov cd33a57f44 [#207] aclsvc: Refactor EACL check 
Resolve funlen linter for CheckEACL method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 13:22:13 +00:00
Dmitrii Stepanov 1f1aed87be [#188] metabase: Refactor object inhume
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for db.Inhume method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 5a66db80c5 [#188] engine: Refactor shard evacuation 
Resolve funlen and gocognit linter for StorageEngine.Evacuate method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 456bc097f7 [#188] engine: Refactor get range from engine 
Resolve funlen linter for StorageEngine.getRange method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 3010ca2649 [#188] engine: Refactor get object from engine 
Resolve funlen linter for StorageEngine.get method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 0739c36a3b [#188] metabase: Refactor object put to metabase 
Resolve funlen linter for db.put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 8273a3dfb2 [#188] blobstor: Refactor blobstor test 
Resolve funlen linter for TestIterate function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Dmitrii Stepanov 594b5821ed [#188] blobstor: Refactor put data to blobovniczas 
Resolve funlen linter for Blobovniczas.Put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 14:50:43 +03:00
Evgenii Stratonikov ee7468daa7 [#205] innerring: Provide `alphabetState` param to `epochTimer`
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Fix NPE, introduced in f09ee27a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-04 10:47:31 +00:00
Dmitrii Stepanov 8e5a0dcf27 [#204] gc: Fix GC handlers start
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-04 06:48:27 +00:00
Evgenii Stratonikov 0948a280fa [#195] morph: use blocking unlimited pool for notifications
ci/woodpecker/push/pre-commit Pipeline failed Details 
With non-blocking pool restricted by 10 in capacity, the probability of
dropping events is unexpectedly big. Notifications are an essential part of the FrostFS,
we should not drop anything, especially new epochs.
```
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z        debug        subscriber/subscriber.go:154        new notification event from sidechain        {"name": "NewEpoch"}
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z        warn        event/listener.go:248        listener worker pool drained        {"chain": "morph", "capacity": 10}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-03 15:58:16 +00:00
Dmitrii Stepanov ece6c820e7 [#199] putsvc: Refactor streamer initialization
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Resolve funlen linter for initTarget method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-03 15:58:11 +00:00
Dmitrii Stepanov 27bdddc48f [#199] putsvc: Refactor put object 
Resolve containedctx linter for streamer and remote target

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-03 15:58:11 +00:00
Dmitrii Stepanov cecea8053a [#199] putsvc: Refactor streamer pool 
Resolve staticcheck linter for putBytesPool

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-03 15:58:11 +00:00
Dmitrii Stepanov 14d894178e [#199] putsvc: Refactor placement iterator 
Resolve funlen linter for iteratePlacement method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-03 15:58:11 +00:00
Pavel Karpy a69c6d1ec9 [#2272] morph: Do not subscribe to events without listening
ci/woodpecker/push/pre-commit Pipeline was successful Details 
It led to a neo-go dead-lock in the `subscriber` component. Subscribing to
notifications is the same RPC as any others, so it could also be blocked
forever if no async listening (reading the notification channel) routine
exists. If a number of subscriptions is big enough (or a caller is lucky
enough) subscribing loop might have not finished subscribing before the
first notification is received and then: subscribing RPC is blocked by
received notification (non)handling and listening notifications routine is
blocked by not finished subscription loop.
That commit starts listening notification channel _before_ any subscription
actions.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-04-03 08:42:41 +00:00
Pavel Karpy aa92f977ef [#181] ir: Do not pay for audit by non-alphabet nodes 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-31 12:53:37 +03:00
Pavel Karpy f09ee27af9 [#181] ir: Do not process container estimations by non-alphabet nodes 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-31 12:53:09 +03:00
Pavel Karpy db5321309d [#181] ir: Do not sync the Alphabet by non-alphabet nodes 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-31 12:52:16 +03:00
Pavel Karpy 44d5412e10 [#181] ir: Do not deposit notary GAS by non-alphabet nodes 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-31 12:51:12 +03:00
Dmitrii Stepanov 56161d39b4 [#168] node: Refactor container services 
Resolve containedctx for remoteLoadAnnounceWriter struct

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-31 09:32:59 +03:00
Dmitrii Stepanov a7c79c773a [#168] node: Refactor node config 
Resolve containedctx linter for cfg

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-31 09:32:59 +03:00
Dmitrii Stepanov 8426d25f4b [#185] ir: Resolve containedctx linter
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 19:10:33 +00:00
Dmitrii Stepanov c1cbe6ff2d [#185] ir: Refactor signature verification 
Resolve funlen linter for verifySignature method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 19:10:33 +00:00
Dmitrii Stepanov aeb4bbc51e [#185] ir: Refactor alphabet update 
Resolve funlen linter for processAlphabetSync method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 19:10:33 +00:00
Dmitrii Stepanov 30c18d46cc [#185] ir: Refactor audit 
Resolve funlen linter for processStartAudit method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 19:10:33 +00:00
Dmitrii Stepanov 6be2688fb4 [#185] ir: Refactor gas emit 
Resolve funlen linter for processEmit method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 19:10:33 +00:00
Dmitrii Stepanov 4d160bd4ab [#185] ir: Refactor ir service creation 
Resolve funlen linter for New function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 19:10:33 +00:00
Dmitrii Stepanov c8a6978563 [#185] ir: Refactor ir start 
Resolve funlen linter for Server.Start method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 19:10:33 +00:00
Dmitrii Stepanov 85fb9e77c4 [#185] ir: Resolve funlen linter 
Resolve funlen linter for GlagoliticLetter.String method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 19:10:33 +00:00
Airat Arifullin 7be5a0fd79 [#189] node: Fix bug with sync/atomic package
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-03-30 14:50:43 +00:00
Dmitrii Stepanov ab32067152 [#183] gc: Fix drop expired locked complex objects
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Do not delete bucket keys during iteration

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 15:33:42 +03:00
Dmitrii Stepanov 9f0bce5c15 [#183] gc: Fix drop expired locked simple objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-30 15:33:42 +03:00
Alejandro Lopez 341fe1688f [#139] test: Add test storage implementation
ci/woodpecker/push/pre-commit Pipeline was successful Details 
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-03-29 14:28:49 +00:00
Pavel Karpy e843e7f090 [#184] ir: Delete unused config reader param
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Not used since notary environments do endless and non-configurable
deposits.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-29 10:34:54 +00:00
Pavel Karpy ba58a77f8c [#184] node, ir: Do not wait for already made notary deposits 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-29 10:34:54 +00:00
Pavel Karpy 3646723ae3 [#184] morph: Make deposit logs INFO 
We have already had and solved plenty of deposit issues and notary balance
is a really important thing. Deserves to be INFO even before the huge logs
severity refactor, happens on an app start only.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-29 10:34:54 +00:00
Pavel Karpy 97e201993b [#184] *: Unify done contexts handling 
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-29 10:34:54 +00:00
Airat Arifullin 221203beeb [#180] node: Refactor panics in unit test
ci/woodpecker/push/pre-commit Pipeline was successful Details 
* Replace panics in unit tests by require.NoError and t.Fatalf

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-03-29 12:39:07 +03:00
Dmitrii Stepanov 91717d4b98 [#176] morph: Resolve funlen linter
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-27 18:49:55 +03:00
Evgenii Stratonikov 382ecae96a [#172] Use ContainersOf() for container list fetching
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Previously we were limited by ~2048 containers because of
neo-go VM limits.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-27 14:32:55 +00:00
Evgenii Stratonikov 9e54646248 [#172] client/container: Support listing containers with `containersOf` 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-27 14:32:55 +00:00
Evgenii Stratonikov bf7d80f44b [#172] morph/client: Support iterators via neo-go session API 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-27 14:32:55 +00:00
Pavel Karpy f3ff9fd251 [#73] morph: Rename vars that collide with package names
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-24 09:42:30 +00:00
Pavel Karpy 533e9f8b75 [#59] morph: Adopt updated `neo-go` client API for subs 
It does not use deprecated methods anymore but also adds more code that
removes. Future refactor that will affect more components will optimize
usage of the updated API.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-24 09:42:30 +00:00
aarifullin 34329d67ff [#86] node: Fix unit test and linter errors
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
 2023-03-23 12:42:58 +03:00
Airat Arifullin 9808dec591 [#86] node: Move testing utils to one package
ci/woodpecker/push/pre-commit Pipeline failed Details 
Move testing utils from tests in local_object_storage package to
unified testutil package

Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
 2023-03-23 08:19:15 +00:00
Denis Kirillov 342e571d89 [#159] Add handle __SYSTEM__ sys attributes
ci/woodpecker/pr/pre-commit Pipeline was successful Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-22 17:35:20 +03:00
Pavel Karpy da8da1c63a [#98] fstree: Do not fail iteration over just removed files
ci/woodpecker/push/pre-commit Pipeline was successful Details 
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-22 07:26:27 +00:00
Dmitrii Stepanov 49234b915e [#155] search-service: Fix search with ST
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Search should return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-22 07:21:09 +00:00
Dmitrii Stepanov 1637a3edce [#155] search-service: Add search with ST test 
In case of session token (ST) with object IDs search should
return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-22 07:21:09 +00:00
Evgenii Stratonikov cbc2efb1d6 [#156] object/get: Make toHeadPrm() pass gocognit
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov 8014fdb21a [#156] metabase: Make freePotentialLocks() pass linter checks 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov fb13902db9 [#156] shard: Make refillMetabase() pass linter checks 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov 3f6b962349 [#156] services/tree: Pass context to replicationWorker() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov 5368c4207a [#156] services/tree: Split syncLoop() in functions 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Evgenii Stratonikov 47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply() 
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-22 07:14:18 +00:00
Pavel Karpy 9a4f40626c [#128] IR: Do not try to emit GAS to nobody
ci/woodpecker/push/full-pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Fix sending GAS to an empty extra wallets receivers list. Also, send GAS to
extra wallets even if netmap is empty.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-21 08:59:46 +00:00
Dmitrii Stepanov 5059dcc19d [#145] shard-gc: Delete expired objects after locks 
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 11:31:08 +03:00
Dmitrii Stepanov 6c4a1699ef [#145] shard-gc: Expired locked unit test 
Added unit test that verifies that GC deletes expired
locked objects in one epoch.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 11:31:08 +03:00
Pavel Karpy 9cd8f7cea0 [#152] IR: Process empty basic incomes
ci/woodpecker/push/full-pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
If network is not configured for basic income earnings, do not distribute
GAS by the Alphabet nodes.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-21 06:59:33 +00:00
Dmitrii Stepanov 44b86bac5a [#148] linter: Add contextcheck linter
ci/woodpecker/push/full-pre-commit Pipeline failed Details
ci/woodpecker/push/pre-commit Pipeline was successful Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov 481a1ca6f3 [#148] linter: Add gocognit linter 
Code with high cognitive complexity is hard intuitively to understand

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov 97c36ed3ec [#148] linter: Add funlen linter 
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov cc8ff015b4 [#148] linter: Add containedctx linter 
Context has to be passed as an argument: https://pkg.go.dev/context

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:52:39 +03:00
Dmitrii Stepanov 2dc86058c3 [#148] memstore: Drop space line 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:52:39 +03:00
Artem Tataurov db3ccd2762 [#128] innerring: Add GAS pouring mechanism for a configurable list of wallets 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-03-20 12:50:05 +00:00
Pavel Karpy abd21f8099 [#136] Revert "[#2260] services/object: Do not assemble object with TTL=1" 
This reverts commit 2567f8020e. It assumes
that assembling logic could break some failover scenarios if request
forwarding is done. However, it also breaks requesting big objects via a
non-container node with TTL=2. Failover has been rechecked without that
commit and no problems were found. Any (if found) other bugs related to
the forwarding and object assembling must be solved more carefully.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-20 10:52:18 +03:00
Pavel Karpy 64bde68fb9 [#67] node: Accept expired locked objects 
Allow replication of any (expired too) locked object. Information about
object locking is considered to be presented on the _container nodes_.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-16 16:22:19 +03:00
Pavel Karpy f006f3b342 [#67] node: Make engine's `IsLocked` public 
It will allow reusing that method in expiration checks.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-16 16:20:45 +03:00
Evgenii Stratonikov 22be532cbd object/put: Persist session token till the end of a session 
Previously a token could've expired in the middle of an object.PUT
stream, leading to upload being interrupted. This is bad, because user
doesn't always now what is the right values for the session token
lifetime. More than that, setting it to a very high value will
eventually blow up the session token database.

In this commit we read the session token once and reuse it for the whole
stream duration.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-16 06:45:50 +00:00
Alejandro Lopez 724debfdcd [#81] node: Add basic read/write benchmarks for substorages 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-03-15 16:37:04 +00:00
Dmitrii Stepanov ac0a278a05 [#85] get-service: Drop unused assemble flag 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Dmitrii Stepanov b8e93d4c08 [#85] get-service: Use assembler to assemble LOB 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Dmitrii Stepanov 07de839f18 [#85] get-service: Fix corrupted chain logic 
Should return an error in case of a broken LOB reference chain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Dmitrii Stepanov 2886b1581b [#85] get-service: Add unit tests 
Add unit tests to cover all assemble statements

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Dmitrii Stepanov 8b9e40a848 [#85] get-service: Add assembler 
Extract assemble logic to assembler

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-15 09:19:45 +03:00
Evgenii Stratonikov 6cd806f998 [#82] services/tree: Save last synchronized height in a persistent storage 
Remember the last synchronized height and use it after service restart.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 11:25:44 +00:00
Evgenii Stratonikov 3e6fd4c611 [#82] pilorama: Allow to store last sync height 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 11:25:44 +00:00
Anton Nikiforov 5ae4446280 [#50] ir: Add Health status 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-03-13 11:24:42 +00:00
Anton Nikiforov 5890cd4d7d [#50] ir: Fix config property name for prometheus 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-03-13 11:24:42 +00:00
Evgenii Stratonikov 05471d3827 [#83] util/autocomplete: Fix deprecated warning 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 07:07:30 +00:00
Evgenii Stratonikov 861e9ab59a [#83] pre-commit: Add initial configuration 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-13 07:07:29 +00:00
Evgenii Stratonikov 6226c3ba86 [#129] policer: Use safer defaults 
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-10 10:59:15 +00:00
Evgenii Stratonikov f2250a316f [#129] tree: Do not remove tree if the netmap is empty 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-10 10:59:15 +00:00
Pavel Karpy f1f3c80dbf [#32] node: Init write-cache asynchronously 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-09 11:07:33 +00:00
Pavel Karpy 381e363a8b [#32] node: Always close general components after testing 
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-03-09 11:07:33 +00:00
Alexey Vanin 20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Anton Nikiforov e9f3c24229 [#65] Use `strings.Cut` instead of `strings.Split*` where possible 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-02-28 13:39:14 +03:00
Dmitrii Stepanov 6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods 
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-02-28 13:36:25 +03:00
Dmitrii Stepanov c3a7039801 [TrueCloudLab/hrw#2] node: Optimize shard hash 
Compute shard hash only once

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-02-28 13:36:25 +03:00
Alejandro Lopez 73bb590cb1 [#64] node: Use pool_size_local and separate pool for local puts 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-02-22 13:43:19 +03:00
Alejandro Lopez cb5468abb8 [#66] node: Replace interface{} with any 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-02-21 16:47:07 +03:00
Denis Kirillov 87e69b9349 [#44] node: Support multiple configs 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-21 10:00:28 +03:00
Pavel Karpy 337049b2ce [#56] node: Allow reading expired locked object 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-21 09:56:57 +03:00
Pavel Karpy 3beef10f89 [#61] node: Do not fetch missing objects 
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 14:47:38 +03:00
Anton Nikiforov 22f3c7d080 [#1868] Reload config for pprof and metrics on SIGHUP 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov 0b61a3c961 [#2260] network/cache: Ignore clients only on `Dial` errors 
The problem is that accidental timeout errors can make us to ignore
other nodes for some time. The primary purpose of the whole ignore
mechanism is not to degrade in case of failover. For this case,
closing connection and limiting the amount of dials is enough.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov bf1e59bb83 [#2260] network/cache: Ignore `context cancelled` errors 
Timeouts on client side should node affect inter-node communication.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov 2567f8020e [#2260] services/object: Do not assemble object with TTL=1 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov d1d123d180 [#2234] writecache: Fix possible panic in `initFlushMarks` 
In case we have many small objects in the write-cache, `indices` should
not be reused between iterations.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov 315141dc2c [#2252] fstree: Allow concurrent writes 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Pavel Karpy b422ac9f94 [#2164] node: Fix multi-client error reporting 
Missing `ReportError` method did not allow casing multi-client interface to
`errorReporter` interface and dropping broken connections.
`replicationClient` embeds that interface, and it is widely used across
node's code. Embedded interface does not allow casting its parent structure
to `errorReporter` and breaks multi client error reporting logic.
Multi-client scheme is extremely hard to maintain, it makes unpredictable
casts and does not allow tracking code flow, so it will be refactored in the
future anyway.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 13:53:27 +03:00
Pavel Karpy 95ee905861 [#2244] node: Fix subscriptions lock 
Subscribing without async listening could lead to a dead-lock in the
`neo-go` client.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 13:53:27 +03:00
Pavel Karpy 07ec51ea60 [#2244] node: Add object address to WC's operations 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 13:53:27 +03:00
Pavel Karpy dbbbef9ddb [#2244] node: Update expired storage ID by WC 
Previously, node could get an "infinite" small object: it could be expired
and thus could not be flushed (update its storage ID) to metabase => could
not be marked as flushed => node never removes such object and repeat all
the cycle one more time. If object exists and is not marked with GC (meta
returns `ErrObjectIsExpired`, not `ObjectNotFound` and not
`ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt
transaction.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov 6fd88a036f [#2241] metrics: Fix request count metrics names 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov 5cb2c5ae62 [#2238] engine: Add test for component initialization failures 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov 427fe276f2 [#2238] shard: Try closing all components 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov c53903ccd0 [#2238] engine: Make `Open` and `Init` similar 
1. Both could initialize shards in parallel.
2. Both should close shards after an error.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00
Evgenii Stratonikov e0309e398c [#2239] writecache: Fix possible deadlock 
LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction.
`View` transaction itself takes `mmapLock` [1], which is lifted after tx
finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`)

When we evict items from LRU cache mutex order is different:
first we take LRU mutex and then execute `Batch` which _does_ take
`mmapLock` in case we need to remap. Thus the deadlock.

[1] 8f4a7e1f92/db.go (L708)

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-02-20 13:53:27 +03:00