56282edf02
[ #166 ] node: Parallelize background tree service sync
...
* Run sync task for nodes in parallel within errgroup worker pool
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 04:43:32 +00:00
4ec69cbbf8
[ #203 ] control: Fix test
...
Fix shard compare loop.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
f32f61df87
[ #203 ] pilorama: Refactor tests
...
Do not pass 0 as channel capacity.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
8908798f59
[ #203 ] node: Resolve unused vars
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
bab11492ad
[ #203 ] node: Resolve never used errors
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
68a2f36636
[ #203 ] morph: Mark depracated methods
...
Skip staticcheck for depracated methods. Will be fixed soon.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
9e2df4b7c7
[ #203 ] node: Fix double imports
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
ab891517de
[ #116 ] node: Fix bug with extra generated files in TestReload
...
* Create testNewEngine in engineWithShards without default opts
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 18:49:34 +03:00
c58ab0c369
[ #193 ] getsvc: Reduce private key requests
...
Get private key only once for request forwaring.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
89924071cd
[ #193 ] getsvc: Edit request forwarder signature
...
Pass context to forwarder direct, without closure.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
6c7b708a98
[ #193 ] getsvc: Refactor get range params creation
...
Resolve funlen linter for toRangePrm function.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
b0786d2e5c
[ #193 ] getsvc: Refactor get params creation
...
Resolve funlen linter for toPrm function.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
f889893216
[ #193 ] getsvc: Refactor head param creation
...
Resolve funlen linter for toHeadPrm method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
91ead04fa4
[ #193 ] getsvc: Resolve funlen linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
1bf21dbb47
[ #193 ] getsvc: Resolve context linters
...
Resolve containedctx and contextcheck linters.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
206458c841
[ #217 ] containersvc: Resolve containedctx linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:00 +00:00
279261ace3
[ #217 ] containersvc: Refactor route passing
...
Resolve containedctx for routeCtx.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:00 +00:00
6f7b6a8813
[ #116 ] node: Improve shard/engine construction in tests
...
* Introduce testEngineWrapper that can be constructed with different options
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 14:36:40 +00:00
d6486d172e
[ #210 ] policier: Refactor nodes processing
...
Resolve funlen linter for processNodes method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
080be5cfcd
[ #210 ] policier: Refactor object placement
...
Resolve containedctx and contextcheck linters.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
23575e1ac0
[ #210 ] policier: Resolve contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
9098d0eec0
[ #211 ] engine: Unify shard mode checks for tree operations
...
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
760af6b912
[ #211 ] fstree: Consider ENOSPC a logical error
...
We already do this for file writing, however directory creation can also
fail.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
d85703a963
[ #208 ] searchsvc: Refactor request forwarding
...
Resolve funlen & gocognit linters for toPrm method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:55 +03:00
0b38419fbf
[ #208 ] searchsvc: Resolve context linters
...
Resolve containedctx and contextcheck linters.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:50 +03:00
5f2a1531fe
[ #208 ] deletesvc: Resolve containedctx linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:40 +03:00
4941926c9d
[ #207 ] aclsvc: Drop outdated tag
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:14 +00:00
585415fa92
[ #207 ] aclsvc: Refactor send checker
...
Resolve funlen linter for putStreamBasicChecker.Send method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:14 +00:00
9ef790f782
[ #207 ] aclsvc: Refactor object headers read
...
Resolve funlen linter for readObjectHeaders method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:13 +00:00
cd33a57f44
[ #207 ] aclsvc: Refactor EACL check
...
Resolve funlen linter for CheckEACL method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:13 +00:00
1f1aed87be
[ #188 ] metabase: Refactor object inhume
...
Resolve funlen linter for db.Inhume method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
5a66db80c5
[ #188 ] engine: Refactor shard evacuation
...
Resolve funlen and gocognit linter for StorageEngine.Evacuate method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
456bc097f7
[ #188 ] engine: Refactor get range from engine
...
Resolve funlen linter for StorageEngine.getRange method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
3010ca2649
[ #188 ] engine: Refactor get object from engine
...
Resolve funlen linter for StorageEngine.get method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
0739c36a3b
[ #188 ] metabase: Refactor object put to metabase
...
Resolve funlen linter for db.put method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
8273a3dfb2
[ #188 ] blobstor: Refactor blobstor test
...
Resolve funlen linter for TestIterate function
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
594b5821ed
[ #188 ] blobstor: Refactor put data to blobovniczas
...
Resolve funlen linter for Blobovniczas.Put method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
ee7468daa7
[ #205 ] innerring: Provide alphabetState param to epochTimer
...
Fix NPE, introduced in f09ee27a
2023-04-04 10:47:31 +00:00
8e5a0dcf27
[ #204 ] gc: Fix GC handlers start
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 06:48:27 +00:00
0948a280fa
[ #195 ] morph: use blocking unlimited pool for notifications
...
With non-blocking pool restricted by 10 in capacity, the probability of
dropping events is unexpectedly big. Notifications are an essential part of the FrostFS,
we should not drop anything, especially new epochs.
```
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z debug subscriber/subscriber.go:154 new notification event from sidechain {"name": "NewEpoch"}
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z warn event/listener.go:248 listener worker pool drained {"chain": "morph", "capacity": 10}
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-03 15:58:16 +00:00
ece6c820e7
[ #199 ] putsvc: Refactor streamer initialization
...
Resolve funlen linter for initTarget method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
27bdddc48f
[ #199 ] putsvc: Refactor put object
...
Resolve containedctx linter for streamer and remote target
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
cecea8053a
[ #199 ] putsvc: Refactor streamer pool
...
Resolve staticcheck linter for putBytesPool
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
14d894178e
[ #199 ] putsvc: Refactor placement iterator
...
Resolve funlen linter for iteratePlacement method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
Pavel Karpy
a69c6d1ec9
[ #2272 ] morph: Do not subscribe to events without listening
...
It led to a neo-go dead-lock in the `subscriber` component. Subscribing to
notifications is the same RPC as any others, so it could also be blocked
forever if no async listening (reading the notification channel) routine
exists. If a number of subscriptions is big enough (or a caller is lucky
enough) subscribing loop might have not finished subscribing before the
first notification is received and then: subscribing RPC is blocked by
received notification (non)handling and listening notifications routine is
blocked by not finished subscription loop.
That commit starts listening notification channel _before_ any subscription
actions.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-03 08:42:41 +00:00
Pavel Karpy
aa92f977ef
[ #181 ] ir: Do not pay for audit by non-alphabet nodes
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:53:37 +03:00
Pavel Karpy
f09ee27af9
[ #181 ] ir: Do not process container estimations by non-alphabet nodes
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:53:09 +03:00
Pavel Karpy
db5321309d
[ #181 ] ir: Do not sync the Alphabet by non-alphabet nodes
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:52:16 +03:00
Pavel Karpy
44d5412e10
[ #181 ] ir: Do not deposit notary GAS by non-alphabet nodes
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:51:12 +03:00
56161d39b4
[ #168 ] node: Refactor container services
...
Resolve containedctx for remoteLoadAnnounceWriter struct
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
a7c79c773a
[ #168 ] node: Refactor node config
...
Resolve containedctx linter for cfg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
8426d25f4b
[ #185 ] ir: Resolve containedctx linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
c1cbe6ff2d
[ #185 ] ir: Refactor signature verification
...
Resolve funlen linter for verifySignature method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
aeb4bbc51e
[ #185 ] ir: Refactor alphabet update
...
Resolve funlen linter for processAlphabetSync method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
30c18d46cc
[ #185 ] ir: Refactor audit
...
Resolve funlen linter for processStartAudit method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
6be2688fb4
[ #185 ] ir: Refactor gas emit
...
Resolve funlen linter for processEmit method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
4d160bd4ab
[ #185 ] ir: Refactor ir service creation
...
Resolve funlen linter for New function
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
c8a6978563
[ #185 ] ir: Refactor ir start
...
Resolve funlen linter for Server.Start method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
85fb9e77c4
[ #185 ] ir: Resolve funlen linter
...
Resolve funlen linter for GlagoliticLetter.String method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
7be5a0fd79
[ #189 ] node: Fix bug with sync/atomic package
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-30 14:50:43 +00:00
ab32067152
[ #183 ] gc: Fix drop expired locked complex objects
...
Do not delete bucket keys during iteration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
9f0bce5c15
[ #183 ] gc: Fix drop expired locked simple objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
341fe1688f
[ #139 ] test: Add test storage implementation
...
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-29 14:28:49 +00:00
Pavel Karpy
e843e7f090
[ #184 ] ir: Delete unused config reader param
...
Not used since notary environments do endless and non-configurable
deposits.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
ba58a77f8c
[ #184 ] node, ir: Do not wait for already made notary deposits
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
3646723ae3
[ #184 ] morph: Make deposit logs INFO
...
We have already had and solved plenty of deposit issues and notary balance
is a really important thing. Deserves to be INFO even before the huge logs
severity refactor, happens on an app start only.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
97e201993b
[ #184 ] *: Unify done contexts handling
...
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
221203beeb
[ #180 ] node: Refactor panics in unit test
...
* Replace panics in unit tests by require.NoError and t.Fatalf
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-29 12:39:07 +03:00
91717d4b98
[ #176 ] morph: Resolve funlen linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 18:49:55 +03:00
382ecae96a
[ #172 ] Use ContainersOf() for container list fetching
...
Previously we were limited by ~2048 containers because of
neo-go VM limits.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
9e54646248
[ #172 ] client/container: Support listing containers with containersOf
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
bf7d80f44b
[ #172 ] morph/client: Support iterators via neo-go session API
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
Pavel Karpy
f3ff9fd251
[ #73 ] morph: Rename vars that collide with package names
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-24 09:42:30 +00:00
Pavel Karpy
533e9f8b75
[ #59 ] morph: Adopt updated neo-go client API for subs
...
It does not use deprecated methods anymore but also adds more code that
removes. Future refactor that will affect more components will optimize
usage of the updated API.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-24 09:42:30 +00:00
aarifullin
34329d67ff
[ #86 ] node: Fix unit test and linter errors
...
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 12:42:58 +03:00
9808dec591
[ #86 ] node: Move testing utils to one package
...
Move testing utils from tests in local_object_storage package to
unified testutil package
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 08:19:15 +00:00
342e571d89
[ #159 ] Add handle __SYSTEM__ sys attributes
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-22 17:35:20 +03:00
Pavel Karpy
da8da1c63a
[ #98 ] fstree: Do not fail iteration over just removed files
...
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133)
2023-03-22 07:26:27 +00:00
49234b915e
[ #155 ] search-service: Fix search with ST
...
Search should return only objects allowed in static session
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-22 07:21:09 +00:00
1637a3edce
[ #155 ] search-service: Add search with ST test
...
In case of session token (ST) with object IDs search should
return only objects allowed in static session
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-22 07:21:09 +00:00
cbc2efb1d6
[ #156 ] object/get: Make toHeadPrm() pass gocognit
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
8014fdb21a
[ #156 ] metabase: Make freePotentialLocks() pass linter checks
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
fb13902db9
[ #156 ] shard: Make refillMetabase() pass linter checks
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
3f6b962349
[ #156 ] services/tree: Pass context to replicationWorker()
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
5368c4207a
[ #156 ] services/tree: Split syncLoop() in functions
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
47e8c5bf23
[ #156 ] pilorama: Remove CIDDescriptor from TreeApply()
...
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
only used in controlled environments.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
Pavel Karpy
9a4f40626c
[ #128 ] IR: Do not try to emit GAS to nobody
...
Fix sending GAS to an empty extra wallets receivers list. Also, send GAS to
extra wallets even if netmap is empty.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-21 08:59:46 +00:00
5059dcc19d
[ #145 ] shard-gc: Delete expired objects after locks
...
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
6c4a1699ef
[ #145 ] shard-gc: Expired locked unit test
...
Added unit test that verifies that GC deletes expired
locked objects in one epoch.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
Pavel Karpy
9cd8f7cea0
[ #152 ] IR: Process empty basic incomes
...
If network is not configured for basic income earnings, do not distribute
GAS by the Alphabet nodes.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-21 06:59:33 +00:00
44b86bac5a
[ #148 ] linter: Add contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
481a1ca6f3
[ #148 ] linter: Add gocognit linter
...
Code with high cognitive complexity is hard intuitively to understand
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
97c36ed3ec
[ #148 ] linter: Add funlen linter
...
Long functions are hard to understand and source of errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
cc8ff015b4
[ #148 ] linter: Add containedctx linter
...
Context has to be passed as an argument: https://pkg.go.dev/context
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
2dc86058c3
[ #148 ] memstore: Drop space line
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
db3ccd2762
[ #128 ] innerring: Add GAS pouring mechanism for a configurable list of wallets
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-03-20 12:50:05 +00:00
Pavel Karpy
abd21f8099
[ #136 ] Revert "[ #2260 ] services/object: Do not assemble object with TTL=1"
...
This reverts commit 2567f8020e
2023-03-20 10:52:18 +03:00
Pavel Karpy
64bde68fb9
[ #67 ] node: Accept expired locked objects
...
Allow replication of any (expired too) locked object. Information about
object locking is considered to be presented on the _container nodes_.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:22:19 +03:00
Pavel Karpy
f006f3b342
[ #67 ] node: Make engine's IsLocked public
...
It will allow reusing that method in expiration checks.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:20:45 +03:00
22be532cbd
object/put: Persist session token till the end of a session
...
Previously a token could've expired in the middle of an object.PUT
stream, leading to upload being interrupted. This is bad, because user
doesn't always now what is the right values for the session token
lifetime. More than that, setting it to a very high value will
eventually blow up the session token database.
In this commit we read the session token once and reuse it for the whole
stream duration.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-16 06:45:50 +00:00
724debfdcd
[ #81 ] node: Add basic read/write benchmarks for substorages
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-15 16:37:04 +00:00
ac0a278a05
[ #85 ] get-service: Drop unused assemble flag
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
b8e93d4c08
[ #85 ] get-service: Use assembler to assemble LOB
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
07de839f18
[ #85 ] get-service: Fix corrupted chain logic
...
Should return an error in case of a broken LOB reference chain.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
2886b1581b
[ #85 ] get-service: Add unit tests
...
Add unit tests to cover all assemble statements
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
8b9e40a848
[ #85 ] get-service: Add assembler
...
Extract assemble logic to assembler
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
6cd806f998
[ #82 ] services/tree: Save last synchronized height in a persistent storage
...
Remember the last synchronized height and use it after service restart.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
3e6fd4c611
[ #82 ] pilorama: Allow to store last sync height
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
5ae4446280
[ #50 ] ir: Add Health status
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 11:24:42 +00:00
5890cd4d7d
[ #50 ] ir: Fix config property name for prometheus
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 11:24:42 +00:00
05471d3827
[ #83 ] util/autocomplete: Fix deprecated warning
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
861e9ab59a
[ #83 ] pre-commit: Add initial configuration
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
6226c3ba86
[ #129 ] policer: Use safer defaults
...
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
f2250a316f
[ #129 ] tree: Do not remove tree if the netmap is empty
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
Pavel Karpy
f1f3c80dbf
[ #32 ] node: Init write-cache asynchronously
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
Pavel Karpy
381e363a8b
[ #32 ] node: Always close general components after testing
...
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
e9f3c24229
[ #65 ] Use strings.Cut instead of strings.Split* where possible
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:39:14 +03:00
6925fb4c59
[ TrueCloudLab/hrw#2 ] node: Use typed HRW methods
...
Update HRW lib and use typed HRW methods to sort shards and nodes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
c3a7039801
[ TrueCloudLab/hrw#2 ] node: Optimize shard hash
...
Compute shard hash only once
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
73bb590cb1
[ #64 ] node: Use pool_size_local and separate pool for local puts
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-22 13:43:19 +03:00
cb5468abb8
[ #66 ] node: Replace interface{} with any
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-21 16:47:07 +03:00
87e69b9349
[ #44 ] node: Support multiple configs
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
Pavel Karpy
337049b2ce
[ #56 ] node: Allow reading expired locked object
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-21 09:56:57 +03:00
Pavel Karpy
3beef10f89
[ #61 ] node: Do not fetch missing objects
...
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 14:47:38 +03:00
22f3c7d080
[ #1868 ] Reload config for pprof and metrics on SIGHUP
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-20 13:53:27 +03:00
0b61a3c961
[ #2260 ] network/cache: Ignore clients only on Dial errors
...
The problem is that accidental timeout errors can make us to ignore
other nodes for some time. The primary purpose of the whole ignore
mechanism is not to degrade in case of failover. For this case,
closing connection and limiting the amount of dials is enough.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
bf1e59bb83
[ #2260 ] network/cache: Ignore context cancelled errors
...
Timeouts on client side should node affect inter-node communication.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
2567f8020e
[ #2260 ] services/object: Do not assemble object with TTL=1
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
d1d123d180
[ #2234 ] writecache: Fix possible panic in initFlushMarks
...
In case we have many small objects in the write-cache, `indices` should
not be reused between iterations.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
315141dc2c
[ #2252 ] fstree: Allow concurrent writes
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
b422ac9f94
[ #2164 ] node: Fix multi-client error reporting
...
Missing `ReportError` method did not allow casing multi-client interface to
`errorReporter` interface and dropping broken connections.
`replicationClient` embeds that interface, and it is widely used across
node's code. Embedded interface does not allow casting its parent structure
to `errorReporter` and breaks multi client error reporting logic.
Multi-client scheme is extremely hard to maintain, it makes unpredictable
casts and does not allow tracking code flow, so it will be refactored in the
future anyway.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
95ee905861
[ #2244 ] node: Fix subscriptions lock
...
Subscribing without async listening could lead to a dead-lock in the
`neo-go` client.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
07ec51ea60
[ #2244 ] node: Add object address to WC's operations
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
dbbbef9ddb
[ #2244 ] node: Update expired storage ID by WC
...
Previously, node could get an "infinite" small object: it could be expired
and thus could not be flushed (update its storage ID) to metabase => could
not be marked as flushed => node never removes such object and repeat all
the cycle one more time. If object exists and is not marked with GC (meta
returns `ErrObjectIsExpired`, not `ObjectNotFound` and not
`ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt
transaction.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
6fd88a036f
[ #2241 ] metrics: Fix request count metrics names
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
5cb2c5ae62
[ #2238 ] engine: Add test for component initialization failures
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
427fe276f2
[ #2238 ] shard: Try closing all components
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
c53903ccd0
[ #2238 ] engine: Make Open and Init similar
...
1. Both could initialize shards in parallel.
2. Both should close shards after an error.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
e0309e398c
[ #2239 ] writecache: Fix possible deadlock
...
LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction.
`View` transaction itself takes `mmapLock` [1], which is lifted after tx
finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`)
When we evict items from LRU cache mutex order is different:
first we take LRU mutex and then execute `Batch` which _does_ take
`mmapLock` in case we need to remap. Thus the deadlock.
[1] 8f4a7e1f92/db.go (L708)
2023-02-20 13:53:27 +03:00
58367e4df6
[ #2232 ] pilorama: Merge in-queue batches
...
To achieve high performance we must choose proper values for both
batch size and delay. For user operations we want to set low delay.
However it would prevent tree synchronization operations to form big
enough batches. For these operations, batching gives the most benefit
not only in terms of on-CPU execution cost, but also by speeding up
transaction persist (`fsync`).
In this commit we try merging batches that are already
_triggered_, but not yet _started to execute_. This way we can still
query batches for execution after the provided delay while also allowing
multiple formed batches to execute faster.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
40822adb51
[ #2213 ] node: Do not return object expired object
...
"Object is expired" means that object is presented in `meta` but it is not
`ObjectNotFound` error. Previous implementation made `shard` search for an
object without `meta` which was an error.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
9afe86ba3e
[ #2212 ] morph: Fix subscription restoration
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
85cf1f47ac
[ #1465 ] node: Prevent process from killing by systemd when shutting down
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-17 12:13:00 +03:00
362f24953a
[ #47 ] shard: Switch container size metric from physical to logical capacity
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-17 12:03:42 +03:00
Pavel Karpy
901d62567d
[ #57 ] node: Broadcast link objects
...
It boosts object assembling by an _average_ container node.
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-17 11:58:27 +03:00
204cd3a11c
[ #31 ] fstree: Optimize treePath
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
dee4498c1e
[ #31 ] fstree: Do not check for a file existence twice
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
abbecf49d6
[ #31 ] fstree: Speedup string-to-address conversion
...
```
name old time/op new time/op delta
_addressFromString-8 1.25µs ±30% 1.02µs ± 6% -18.49% (p=0.000 n=9+9)
name old alloc/op new alloc/op delta
_addressFromString-8 352B ± 0% 256B ± 0% -27.27% (p=0.000 n=9+10)
name old allocs/op new allocs/op delta
_addressFromString-8 6.00 ± 0% 4.00 ± 0% -33.33% (p=0.000
n=10+10)
```
Also, assure compiler that `s` doesn't escape:
Before this commit:
```
./fstree.go:74:24: leaking param: s
./fstree.go:90:6: moved to heap: addr
```
After this commit:
```
./fstree.go:74:24: s does not escape
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
ab21d90cfb
[ #1794 ] shard: Add increasing case for the payload size metric
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-09 13:30:23 +03:00
