3221 commits

Author	SHA1	Message	Date
Evgenii Stratonikov	4f413fe86e	[#1] treesvc: Properly check for secure transport ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-28 12:13:20 +00:00
Evgenii Stratonikov	cab51c8cbe	[#1] metabase: Rename blindlyProcess() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-28 12:13:20 +00:00
Evgenii Stratonikov	f4c71cea65	[#1] *: Replace outdated FIXME/TODO links ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-28 12:13:20 +00:00
Evgenii Stratonikov	0c866f62c5	[#473] placement: Fix backwards compatibility for copies_number ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-27 11:13:16 +03:00
Dmitrii Stepanov	43d263c3d5	[#428] linter: Fix unkeyed assignment ... Thanks to gopls. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-26 13:29:39 +00:00
Dmitrii Stepanov	cac4ed93d6	[#428] engine: Add low_mem config parameter ... Concurrent initialization in case of the metabase resync leads to high memory consumption and potential OOM. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-26 13:29:39 +00:00
Evgenii Stratonikov	4bf345225c	[#447] pilorama: Use named constant for the key size ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-26 07:42:05 +00:00
Evgenii Stratonikov	b4ce0b0412	[#447] pilorama: Do not undo log for create ops ... ``` goos: linux goarch: amd64 cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ new │ │ sec/op │ sec/op vs base │ Create-8 36.48µ ± 11% 30.34µ ± 14% -16.84% (p=0.000 n=10) │ old │ new │ │ B/op │ B/op vs base │ Create-8 43.01Ki ± 4% 37.78Ki ± 5% -12.15% (p=0.000 n=10) │ old │ new │ │ allocs/op │ allocs/op vs base │ Create-8 166.0 ± 3% 146.0 ± 3% -12.05% (p=0.000 n=10) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-26 07:42:05 +00:00
Evgenii Stratonikov	dd3874eff1	[#447] pilorama: Add benchmark for create ops ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-26 07:42:05 +00:00
Anton Nikiforov	71889234b7	[#449] tree: Allow reading requests signed by keys from allow list ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-06-23 14:44:19 +03:00
Evgenii Stratonikov	40eae22109	[#460] services/util: Remove CreateRequestStreamer ... There is no need in a wrapper with many from-`interface{}` conversions. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-21 17:08:04 +03:00
Evgenii Stratonikov	a64dc9ad70	[#460] services/util: Remove HandleServerStreamRequest ... There is no need in a wrapper with many from-`interface{}` conversions. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-21 17:08:01 +03:00
Evgenii Stratonikov	167a67f0b8	[#460] services/util: Remove HandleUnaryRequest ... There is no need in a wrapper with many from-`interface{}` conversions. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-21 17:07:56 +03:00
Evgenii Stratonikov	785d81a68a	[#460] services/object: Reduce distibutedTarget memory footprint ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-21 17:07:50 +03:00
Dmitrii Stepanov	4d48377cec	[#459] blobovniczatree: Fix get error ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 16:44:53 +03:00
Dmitrii Stepanov	03aa210145	[#373] metrics: Move labels to consts ... To unify label naming all lable keys and other consts are moved to one file. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	b5d9f4a285	[#373] metrics: Add pilorama metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	e89fa110c7	[#373] metrics: Add metabase metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	af608da952	[#373] metrics: Add blobovnizca metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	56f320dd85	[#373] metrics: Add blobstor metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	16a142cd0c	[#373] metrics: Add FSTree metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	d8ecc69d00	[#373] local storage: Pass parent ID ... This is required to add shard ID as metric label. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	d5aaec1107	[#373] pilorama: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	059e9e88a2	[#373] metabase: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	f54cc0b607	[#373] blobstor: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	8318d90ad0	[#373] blobovniczatree: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	3ae3c8dfdb	[#373] fstree: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	a8526d45e9	[#373] blobovnizca: Add missed/fix tracing spans ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	028d4a8058	[#373] blobovnicza: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	01a0c97760	[#453] engine: Set Disabled mode to deleted shard ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-20 12:04:07 +03:00
Anton Nikiforov	50caa388b0	[#303] ir: Use pub key when validate container deletion ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-06-20 08:02:48 +00:00
Evgenii Stratonikov	69df0d21c2	[#446] engine: Move to read-only on blobstor errors ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-16 14:53:32 +03:00
Evgenii Stratonikov	fe01781811	[#446] los: Wrap SSD errors in a separate type ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-16 14:53:31 +03:00
Evgenii Stratonikov	20b84f183a	[#446] engine: Simplify logs for shard mode change ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-16 14:51:29 +03:00
Dmitrii Stepanov	69b788a90b	[#424] metrics: Rename mode to mode_info ... To follow best practice. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 18:39:49 +03:00
Dmitrii Stepanov	26b305f82b	[#424] morph: Fix cache metrics ... Use separate morph cache metrics for node and IR Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 18:26:19 +03:00
Dmitrii Stepanov	4449006862	[#424] metrics: Use mode value as metric value for shard ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 18:26:19 +03:00
Dmitrii Stepanov	847732605c	[#424] metrics: Rename counter to total ... Suffix total should be used for a unit-less accumulating count. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:53:32 +03:00
Dmitrii Stepanov	c348ae35b0	[#424] metrics: Drop embedded metrics ... It was not obvious where metrics are used. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:53:32 +03:00
Dmitrii Stepanov	1b364d8cf4	[#424] metrics: Refactor engine metrics ... Use histogram vector to measure request duration. Fix naming like in Prometheus best practice. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:53:32 +03:00
Dmitrii Stepanov	c8023a9c8d	[#424] morphcache: Use labels for method duration ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:53:32 +03:00
Dmitrii Stepanov	85deb12f4d	[#424] writecache: Drop metrics when close ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:52:40 +03:00
Dmitrii Stepanov	07f155ac77	[#424] metrics: Use labels for writecache methods and operations ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:52:40 +03:00
Dmitrii Stepanov	71bbeddb64	[#424] metrics: Drop unused arg ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:51:40 +03:00
Alejandro Lopez	fb8fee0c8e	[#442] Use strconv.FormatBool instead of untyped conversion ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-06-14 11:08:36 +00:00
Alejandro Lopez	344d6b2ae1	[#xx] Fix invalid log metric namespace identifier ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-06-13 15:36:34 +03:00
Alejandro Lopez	4887f489a1	[#17] Add morph client metrics ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-06-13 14:06:04 +03:00
Airat Arifullin	90e9247b69	[#371] ir: Add morph cache metrics ... Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-06-13 10:10:13 +00:00
Dmitrii Stepanov	957a43a124	[#266] services/tree: Add sync check ... Do not accept requests until initial sync is finished. `Apply` is deliberately left out -- we don't want to miss anything new. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-13 10:00:45 +00:00
Dmitrii Stepanov	e69a1e8482	[#266] services/tree: Return operation log up to some height ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-13 10:00:45 +00:00
Dmitrii Stepanov	2541d319de	[#266] pilorama: Allow to get current tree height ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-13 10:00:45 +00:00
Alejandro Lopez	0c40d98f7a	[#375] Add log metrics ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-06-13 09:52:45 +00:00
Dmitrii Stepanov	0400153b7d	[#424] object: Do not store large slices in pool ... Dynamically growing an unbounded buffers can cause a large amount of memory to be pinned and never be freed. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-13 07:19:55 +00:00
Evgenii Stratonikov	41ab4d070e	[#423] *: Use hrw.StringHash() where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-13 07:18:25 +00:00
Dmitrii Stepanov	9bc1a25c07	[#434] metrics: Fix writecache duration ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-13 07:18:01 +00:00
Dmitrii Stepanov	5b75432ca2	[#431] metrics: Add missed label ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-07 13:51:36 +00:00
Anton Nikiforov	263c6fdc50	[#372] node: Add metrics for the error counter in the engine ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-06-07 13:04:47 +00:00
Evgenii Stratonikov	63473d0806	[#417] go.mod: Update neo-go ... Update to master, because after the API update notary signer is being incorrectly formed. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-02 11:43:48 +00:00
Dmitrii Stepanov	55c28fd5f4	[#412] cache: Pass DialOptions to create new conn ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-01 13:23:11 +00:00
Dmitrii Stepanov	dcdfb6ed41	[#412] node: Use observability interceptors ... Use metrics and tracing interceptors. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-01 13:23:11 +00:00
Dmitrii Stepanov	c09144ecf1	[#412] node: Replace metrics package ... Use observability module. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-01 13:23:11 +00:00
Dmitrii Stepanov	74578052f9	[#412] node: Replace tracing package ... Use observability module. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-01 13:23:11 +00:00
Alejandro Lopez	f7c4c07453	[#xx] Create innerring metrics before metrics usage ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-06-01 10:07:48 +03:00
Evgenii Stratonikov	4476a1dbaf	[#413] morph/client: Fix govet warnings ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-31 15:38:17 +03:00
Dmitrii Stepanov	dbf41391b5	[#401] engine: Extend evacuation logs ... Add operation-tag to logger. Log evacuation results. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-31 13:24:30 +03:00
Dmitrii Stepanov	3220c4df9f	[#376] metrics: Add GC metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-31 10:22:12 +00:00
Alejandro Lopez	faca861451	[#411] Remove unnecessary pointers for sync objects ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-31 10:19:14 +00:00
Alejandro Lopez	ebcc8afbee	[#374] Add inner-ring event metrics ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-31 10:11:48 +00:00
Anton Nikiforov	8dcd06c587	[#394] node: Use Context in Blobovniczas.Iterate() ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-31 10:09:18 +00:00
Anton Nikiforov	365a7ca0f4	[#366] node: Stop GC once termination signal received ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-29 09:35:08 +03:00
Anton Nikiforov	802168c0c6	[#364] node: Stop flushing big object when termination signal received ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-26 16:46:58 +03:00
Dmitrii Stepanov	271a56c2ab	[#395] metrics: Drop redundant metrics ... HistogramVec already has labeled counter. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-26 13:39:37 +00:00
Alejandro Lopez	bc34fee6a7	[#370] Add tree service metrics ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-26 13:39:12 +00:00
Evgenii Stratonikov	f2e5dead7e	[#398] pilorama: Disallow applying same operations ... 1. In redo() we save the old state. 2. If we do redo() for the same operation twice, the old state will be overritten with the new one. 3. This in turn affects undo() and subsequent isAncestor() check. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-26 12:14:29 +00:00
Dmitrii Stepanov	20a489bdb5	[#393] gc: Use defer to mark handler done ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-26 12:14:02 +00:00
Dmitrii Stepanov	9119199f6e	[#397] cli: Fix evacuation method names ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-26 11:28:30 +03:00
Dmitrii Stepanov	2613351008	[#387] gc: Cancel GC is change mode requested ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-25 09:38:16 +03:00
Dmitrii Stepanov	2ce43935f9	[#312] metrics: Add writecache metrcis ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 10:18:39 +00:00
Dmitrii Stepanov	d212d908b5	[#312] wc: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 10:18:39 +00:00
Dmitrii Stepanov	4503a61997	[#312] wc: Delete unused Iterate method ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 10:18:39 +00:00
Dmitrii Stepanov	53a1b15693	[#338] ir: Drop notaryless code from governance ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 08:44:47 +00:00
Dmitrii Stepanov	0ab589dd52	[#338] ir: Always enable notary on sidechain ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 08:44:47 +00:00
Dmitrii Stepanov	81718afb39	[#338] ir: Drop named named put fee ... Named put fee value used only when notary disabled. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 08:44:47 +00:00
Dmitrii Stepanov	656fd7f376	[#338] ir: Drop notaryless code from netmap ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 08:44:47 +00:00
Dmitrii Stepanov	fb708b3a2d	[#338] ir: Drop container notaryless code ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 08:44:47 +00:00
Evgenii Stratonikov	4b768fd115	[#381] *: Move to sync/atomic ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-23 08:18:01 +03:00
Evgenii Stratonikov	ff570847a4	[#381] go.mod: Update bbolt ... Adopt new `ForEachBucket` function where possible. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-22 11:49:14 +03:00
Evgenii Stratonikov	35c9b6b26d	[#314] writecache: remove objects right after they are flushed ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-19 09:37:05 +00:00
Pavel Karpy	bf79d06f03	[#314] writecache: Do not lose small objects on disk errors ... Do return error if an object could not been stored on WC's disk. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-05-19 09:37:05 +00:00
Pavel Karpy	9e56592be3	[#314] writecache: Simplify background workers naming ... Also, drop not used arg. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-05-19 09:37:05 +00:00
Dmitrii Stepanov	f7c0b50d70	[#329] cli: Add async evacuate commands ... Add start, stop evacuate and evacuate status commands. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-19 08:43:52 +00:00
Dmitrii Stepanov	e4889e06ba	[#329] node: Make evacuate async ... Now it's possible to run evacuate shard in async. Also only one evacuate process can be in progress. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-19 08:43:52 +00:00
Dmitrii Stepanov	100b1b5128	[#329] node: Add async evacuate proto methods ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-19 08:43:52 +00:00
Alejandro Lopez	13a7a90101	[#355] Increase tree svc client cache size to test hypotheses ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-17 14:11:46 +03:00
Anton Nikiforov	6f47c75e43	[#125] ir: Set extra wallets on SIGHUP ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-16 12:47:43 +00:00
Dmitrii Stepanov	869fcbf591	[#332] gc: Fix expired complex object deletion ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-16 12:44:57 +00:00
Dmitrii Stepanov	ab07bad33d	[#332] gc: Add complex object unit test ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-16 12:44:57 +00:00
Alejandro Lopez	a6ee7a3087	[#324] Add replicator metrics ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-12 11:00:17 +00:00
Evgenii Stratonikov	6055b18362	[#342] morph: Remove unused toStackParameter() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 13:24:49 +03:00
Evgenii Stratonikov	02c02974b3	[#309] locode: Parallelize DB generation ... For v0.4.0 release: Before: ``` Executed in 571.64 secs fish external usr time 283.07 secs 744.00 micros 283.07 secs sys time 8.41 secs 179.00 micros 8.41 secs ``` After: ``` Executed in 54.23 secs fish external usr time 418.65 secs 1.01 millis 418.65 secs sys time 0.61 secs 0.25 millis 0.60 secs ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:59:13 +00:00
Evgenii Stratonikov	c62025c836	[#321] metabase/test: execute tests in parallel ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	945454f60c	[#321] engine/test: Execute tests in parallel ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	4578d00619	[#321] shard/test: Execute tests in parallel ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	d35e4c389f	[#321] shard/test: Parallelize TestWriteCacheObjectLoss ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	969bfb603f	[#321] shard/test: Parallelize TestShard_List ... ``` go test -count=1 -run TestShard_List -race . Before: 2.492s After: 0.109s ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	47b0ec33c3	[#337] netmap: Remove unused events ... Done in TrueCloudLab/frostfs-contract#16. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:19:38 +00:00
Evgenii Stratonikov	b480df4985	[#337] container: Remove unused events ... Done in TrueCloudLab/frostfs-contract#16. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:19:38 +00:00
Anna Shaleva	bcdb0f330d	[#337] morph: Completely remove fallbackTime from client cfg ... It's unused and not needed, default fallback lifetime is set by Notary actor. Signed-off-by: Anna Shaleva <anna@nspcc.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:19:38 +00:00
Anna Shaleva	ddcc156ecc	[#337] morph: Use Notary Actor for notary requests ... Signed-off-by: Anna Shaleva <anna@nspcc.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:19:38 +00:00
Evgenii Stratonikov	35fdf6f315	[#337] morph: Move subscription logic to subscriber ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:19:38 +00:00
Roman Khimov	a5f118a987	[#337] subscriber: Drop unused UnsubscribeForNotification ... It's not really needed, closing the connection works fine when exiting and normally the app doesn't need to unsubscribe at all. Signed-off-by: Roman Khimov <roman@nspcc.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:19:38 +00:00
Anton Nikiforov	800eb5e983	[#125] ir: Reconfigure pprof and metrics on SIGHUP ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-11 12:38:34 +03:00
Dmitrii Stepanov	a181c9e434	[#332] gc: Add additional logging ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-10 17:34:40 +03:00
Alejandro Lopez	90799497d3	[#114] Add remove-node IR control command ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-10 14:31:44 +00:00
Dmitrii Stepanov	ea10abb42a	[#296] morph: Add parser unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-10 09:40:26 +03:00
Dmitrii Stepanov	1309622b20	[#296] morph: Add listener unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-10 09:40:26 +03:00
Evgenii Stratonikov	b2ffd7df53	[#291] object: Use PayloadSizeLimiter from SDK ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-05 19:07:06 +03:00
Evgenii Stratonikov	35ea207df6	[#291] object: Split validating target in two ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-05 19:07:06 +03:00
Pavel Karpy	ee58b390bb	[#221] node: Allow using vector copies_number ... Also, take into account that value in general (it was not used before at all). Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-05-05 16:07:13 +00:00
Alejandro Lopez	973af12854	[#327] tests: replace os.MkdirTemp with t.TempDir ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-05 11:24:51 +00:00
Anton Nikiforov	cedd07bbc8	[#304] morph: Iterate endpoints when create ws client in constructor ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-05 10:07:30 +03:00
Pavel Karpy	479c5a65e1	[#322] node: Fix tree svc panic ... If a connection has not been established earlier, it stores `nil` in LRU cache. Cache eviction tries to close every connection (even a `nil` one) and panics but not crash the app because we are using pools. That ugly bug also leads to a deadlock where `Unlock` is not called via `defer` func (and that is the way I found it). Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-05-04 19:48:51 +03:00
Dmitrii Stepanov	2f6757c828	[#311] ir: Fix data race in unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-04 16:59:15 +03:00
Evgenii Stratonikov	a1b4ba9980	[#209] compression: Do not store uncompressible data ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-03 16:28:29 +03:00
Dmitrii Stepanov	529d0bc710	[#302] tree: Drop unused ctx ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 16:54:06 +03:00
Dmitrii Stepanov	eca5c210dd	[#299] evacuate: Add context cancel checks ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 15:55:30 +03:00
Dmitrii Stepanov	b939e4e5c5	[#299] ir: Drop unused structs ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 14:05:35 +03:00
Evgenii Stratonikov	235fe84ea3	[#298] innerring: Fix broken tests ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-02 11:52:29 +03:00
Dmitrii Stepanov	d00b1c0d29	[#280] ir: Add netmap processor unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	fb5dcc15d2	[#280] ir: Add governance processor unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	31b4da225a	[#280] ir: Add frostfs processor unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	5010b35466	[#280] ir: Add container processor unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	686f01bce5	[#280] ir: Add balance processor unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	e89fa7f69f	[#280] ir: Add alphabet processor unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	53693071de	[#280] ir: Add block timer unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	e70f808dc3	[#280] ir: Add parser unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	2dbe382b5f	[#280] ir: Add state unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	500611f3c8	[#280] ir: Add indexer tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	3b64dffda2	[#280] ir: Add fee config tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 08:48:07 +00:00
Dmitrii Stepanov	22d47376a6	[#277] getsvc: Refactor errors ... Move errors to separate files. Use zap.Error for error logging. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-28 14:03:12 +00:00
Dmitrii Stepanov	45438e7b06	[#277] getsvc: Rename and reorder code ... Rename execCtx to request. Move code to appropriate files. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-28 14:03:12 +00:00
Dmitrii Stepanov	1440450606	[#277] getsvc: Drop cyclic struct dependency ... Drop cyclic dependency between execCtx and Service. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-28 14:03:12 +00:00
Dmitrii Stepanov	591c4e7d50	[#277] getsvc: Move headOnly to request params ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-28 14:03:12 +00:00
Dmitrii Stepanov	265d2326a0	[#277] getsvc: Extract remote storage ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-28 14:03:12 +00:00
Dmitrii Stepanov	30e1b62b67	[#277] getsvc: Fix service deps ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-28 14:03:12 +00:00
Dmitrii Stepanov	8fc082b688	[#277] getsvc: Do not return status error ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-28 14:03:12 +00:00
Evgenii Stratonikov	f368ccbdf0	[#248] morph: Remove audit client ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 17:00:43 +03:00
Evgenii Stratonikov	a9c4ba62c3	[#248] metabase: Remove storage group bucket ... Backwards compatible change, so no version increase. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 17:00:43 +03:00
Evgenii Stratonikov	586f5986bc	[#248] core: Remove storage group ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 17:00:43 +03:00
Evgenii Stratonikov	f1ea8fec93	[#248] object_manager: Remove storage group ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 17:00:43 +03:00
Evgenii Stratonikov	7f49f07255	[#248] services: Remove audit service ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 17:00:43 +03:00
Evgenii Stratonikov	8879c6ea4a	[#248] innerring: Remove audit and settlement code ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 17:00:43 +03:00
Evgenii Stratonikov	f73ac6e02d	[#290] control: Use generics for response wrappers ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 10:57:24 +03:00
Alejandro Lopez	ff25521204	[#270] Add IR epoch tick control call ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-04-28 07:57:00 +00:00
Evgenii Stratonikov	58f1ba4b51	[#288] pilorama: Add missing operation in log ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Evgenii Stratonikov	daa26f6e9b	[#288] pilorama/test: Check operation order for TreeGetByPath() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Evgenii Stratonikov	291f9e809a	[#288] pilorama: Remove getMeta() wrapper ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Evgenii Stratonikov	0045f1bcd4	[#288] pilorama: Use more descriptive names for memory tree ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Evgenii Stratonikov	f856ad7480	[#288] pilorama: Remove childMap from memory forest ... Memory forest is here to check the correctness of boltdb optimized implementation. Let's keep it simple. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Airat Arifullin	ada081dfd5	[#19] node: Make policier read shards concurrently ... * Introduce ListWithMultiCursor that simultaneously reads objects from different shards Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-27 11:11:47 +03:00
Evgenii Stratonikov	1f4061c0e2	[#285] blobonicza: Optimize upperPowerOfTwo() ... The real reason is this: ``` pkg/local_object_storage/blobovnicza/sizes.go:36:69 revive empty-block: this block is empty, you can remove it ``` Didn't want to make this function longer or to add `nolint`, thus this change. To justify: ``` UpperBound/size=1-8 0.4924n ± 1% 0.2472n ± 2% -49.80% (p=0.000 n=10) UpperBound/size=1023-8 0.4936n ± 3% 0.2442n ± 1% -50.52% (p=0.000 n=10) UpperBound/size=66560-8 0.8201n ± 2% 0.2436n ± 1% -70.29% (p=0.000 n=10) UpperBound/size=41943040-8 6.6900n ± 5% 0.2432n ± 0% -96.36% (p=0.000 n=10) geomean 1.075n 0.2446n -77.24% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-26 17:35:11 +03:00
Evgenii Stratonikov	dfe4ada838	[#285] lint: Resolve revive/if-return ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-26 17:35:08 +03:00
Evgenii Stratonikov	f07e2d4812	[#285] lint: Fix revive/unused-parameter ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-26 17:35:04 +03:00
Airat Arifullin	9d01029733	[#166] node: Parallelize background tree service sync by batching ... * Merge operations Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-26 10:17:56 +00:00
Airat Arifullin	299b24b974	[#166] node: Parallelize background tree service sync by batching ... * Concurrently dispatch TreeApply operations for batching in forest Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-26 10:17:56 +00:00
Denis Kirillov	dce5924a89	[#229] services/tree: Use bearer owner as signer ... Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>	2023-04-26 10:23:33 +03:00
Alex Vanin	89530534a1	[#229] service/tree: Disable container owner check in tree service ... Signed-off-by: Alex Vanin <a.vanin@yadro.com>	2023-04-26 10:23:33 +03:00
Alex Vanin	c04f6c5e59	[#229] acl: Allow Impersonate ... Signed-off-by: Alex Vanin <a.vanin@yadro.com>	2023-04-26 10:23:33 +03:00
Evgenii Stratonikov	6fef2726b8	[#164] metrics: Allow to export metrics description ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-24 09:01:13 +00:00
Evgenii Stratonikov	4ade5339da	[#164] metrics: Rename metrics.go -> node.go ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-24 09:01:13 +00:00
Evgenii Stratonikov	015d62425b	[#164] metrics: Fill local registry explicitly ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-24 09:01:13 +00:00
Pavel Karpy	09938a9841	Revert "[#262] meta: Do not return old expired objects" ... This reverts commit 3d23b087 Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-20 19:22:55 +03:00
Pavel Karpy	e9461686b8	[#274] wc: Resolve possible deadlock ... If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is called immediately) there is a race and a deadlock that do not allow finish (and start, in fact) an initialization routine because of taken `modeMtx` and also do not allow finish `Close` call because of awaiting initialization finish. So do stop initialization _before_ any mutex is taken. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-20 19:22:54 +03:00
Pavel Karpy	6b6f33ed71	[#274] wc: Make wait groups work more explicit ... Do not run routine that calls `wg.Done()` inside, it is hard to read. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-20 19:22:54 +03:00
Pavel Karpy	4f5f832137	[#268] notary_preparator: Actualize notary requests parsing ... After 75d7891ca1 `neo-go` does claim that an empty invocation script is the only way to fill missing signature for unsigned notary requests. The new notary actor does it that way and, therefore, breaks notary request parsing by the Alphabet because of skipping any request that is not filled with a dummy (64 zeros) invocation script. Support both way. The "Dummy" approach will be dropped later. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-20 10:29:34 +03:00
Airat Arifullin	6c90bb87f1	[#118] node: add ctx for unit tests for blobstor ... Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-19 18:09:33 +03:00
Pavel Karpy	3d23b08773	[#262] meta: Do not return old expired objects ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-19 13:37:38 +00:00
Airat Arifullin	13c8afcb02	[#118] node: add unit concurrent tests for blobstor ... Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-19 10:22:50 +00:00
Pavel Karpy	20cd080323	[#255] write-cache: Fix init race condition ... Do not use WC's internals in the initialization routines without mode protection. WC should be able to change its mode even if the initialization is not finished yet. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-19 09:29:07 +00:00
Dmitrii Stepanov	3d43b0f7f9	[#265] node: Fix after SDK & API-Go version up ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-18 12:09:19 +03:00
Dmitrii Stepanov	a358255c1b	[#265] node: Drop unused ... Resolve unused linter. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-18 12:09:16 +03:00
Dmitrii Stepanov	f07d4158f5	[#249] node: Drop subnet from IR and morph ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-17 09:46:02 +03:00
Pavel Karpy	262c9c2b93	[#256] blobovniczaTree: Make Exists test stable ... Corrupt and request _the same_ file. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-14 16:46:24 +03:00
Evgenii Stratonikov	0b42a00a60	[#254] innerring: Remove unused TimersHandlers() method from processors ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-14 15:34:15 +03:00
Evgenii Stratonikov	8466894fdf	[#250] control: remove DumpShard and RestoreShard RPC ... We have `Evacuate` with a cleaner interface. Also, remove them from CLI and engine. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-14 12:28:49 +00:00
Pavel Karpy	070154d506	[#247] client: Drop reputation related RPCs ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-14 14:54:28 +03:00
Pavel Karpy	beabed788c	[#247] network_config: Drop reputation ... Drop the code that was expected to work with global reputation network parameters. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-14 14:53:22 +03:00
Pavel Karpy	8799138fcb	[#247] morph: Drop reputation contract ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-14 14:44:14 +03:00
Pavel Karpy	960e3c219e	[#247] config, doc: Drop reputation references ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-14 14:44:14 +03:00
Pavel Karpy	560f73ab7e	[#247] node, ir: Drop reputation related code ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-14 14:44:14 +03:00
Dmitrii Stepanov	6121b541b5	[#242] treesvc: Add tracing spans ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-14 10:25:53 +00:00
Dmitrii Stepanov	d62c6e4ce6	[#242] node: Add tracing spans ... Add tracing spans for PUT requests. Add tracing spans for DELETE requests. Add tracing spans for SELECT requests. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-14 10:25:53 +00:00
Dmitrii Stepanov	200fc8b882	[#242] put: Pass context to relay function ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-14 10:25:53 +00:00
Anton Nikiforov	995db117d0	[#238] node: Read cfg from dir even if cfg file not set ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-04-14 10:19:10 +00:00
Alejandro Lopez	41eb3129ae	[#139] Refactor blobovnicza exist test to not use chmod ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-04-14 12:16:14 +03:00
Anton Nikiforov	299b6a6938	[#100] adm: Use netmap constants from pkg ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-04-14 05:12:50 +00:00
Evgenii Stratonikov	cffcc7745e	[#240] logs: Factor out common service log messages ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-14 05:06:09 +00:00
Evgenii Stratonikov	0e31c12e63	[#240] logs: Move log messages to constants ... Drop duplicate entities. Format entities. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-14 05:06:09 +00:00
Roman Khimov	d29b13454f	[#239] morph/client: Simplify code interacting with magic numbers ... It can't be uint64 in fact, but this error is buried deeply in the NetworkInfo API structure, so we're not touching MagicNumber() for now. Signed-off-by: Roman Khimov <roman@nspcc.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-13 13:15:30 +00:00
Roman Khimov	f41ad9d419	[#239] morph/client: Recreate actor/wrappers in SetGroupSignerScope ... That's the reason #2230 and #2263 were not detected earlier, we actually had Global scope being used before reconnection to RPC node. Signed-off-by: Roman Khimov <roman@nspcc.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-13 13:15:30 +00:00
Roman Khimov	be4df989e5	[#239] morph/client: Deduplicate signers in Client a bit ... One signer in the cfg is enough. Signed-off-by: Roman Khimov <roman@nspcc.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-13 13:15:30 +00:00
Roman Khimov	96b38f7e86	[#239] morph/client: Add CalledByEntry into the "grouped" scope ... Fixes #2230, fixes #2263. CustomGroups are nice while we're only calling NeoFS contracts, but it doesn't work at all for standard ones like GAS or Notary. Signed-off-by: Roman Khimov <roman@nspcc.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-13 13:15:30 +00:00
Evgenii Stratonikov	04727ce1d6	Revert "[#135] signature: Add tracing" ... This reverts commit 5778980252.	2023-04-12 16:57:02 +03:00
Evgenii Stratonikov	08769f413f	Revert "[#135] acl: Add tracing spans" ... This reverts commit b2ca730547.	2023-04-12 16:54:13 +03:00
Evgenii Stratonikov	5d2affa5cd	testutil: Fix linter warning ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-12 11:00:02 +03:00
Dmitrii Stepanov	5778980252	[#135] signature: Add tracing ... Add tracing to verify request and sign response. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-12 06:52:00 +00:00
Dmitrii Stepanov	b2ca730547	[#135] acl: Add tracing spans ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-12 06:52:00 +00:00
Dmitrii Stepanov	0920d848d0	[#135] get-object: Add tracing spans ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-12 06:52:00 +00:00
Dmitrii Stepanov	5af9f58469	[#135] tracing: Add tracing to node gRPC endpoints ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-12 06:52:00 +00:00
Evgenii Stratonikov	c4865783fc	[#236] blobstor/test: Prefill storage in parallel in read benchmark ... `blobovniczatree` takes a really long time to prefill, because each batch takes at least 10ms, so for 10k iterations we have at least 100s of prefill. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-11 20:52:11 +03:00
Evgenii Stratonikov	6ad5c38225	[#236] testutil: Use random object id in RandObjGenerator ... Before this commit it was like this: ``` BenchmarkSubstorageWritePerf/memstore-rand10-8 227425 4859 ns/op BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8 perf_test.go:165: writing entry: file exists perf_test.go:165: writing entry: file exists perf_test.go:165: writing entry: file exists BenchmarkSubstorageWritePerf/fstree-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8 perf_test.go:165: writing entry: file exists perf_test.go:165: writing entry: file exists perf_test.go:165: writing entry: file exists ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-11 20:52:04 +03:00
Evgenii Stratonikov	c85a0bc866	[#236] blobstor/test: Reduce test descriptions ... I tried to add 4 more tests and suddenly, it became harder to navigate in code. Move directory creation in a common function. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-11 20:52:04 +03:00
Dmitrii Stepanov	93eba19a8e	[#223] objectsvc: Refactor split-tree traverse ... Resolve funlen & gocognit linters for traverseSplitChain method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-10 14:16:34 +03:00
Dmitrii Stepanov	2ed9fd3f94	[#223] objectsvc: Refactor request parameters ... Resolve containedctx linter for commonPrm. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-10 14:16:34 +03:00
Dmitrii Stepanov	ccf8463e69	[#223] controlsvc: Drop unnecessary nolint ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-10 14:16:34 +03:00
Dmitrii Stepanov	ae86cda58c	[#223] sg: Refactor storage group parameters ... Resolve containedctx linter for SearchSGPrm and GetSGPrm structs. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-10 14:16:34 +03:00
Dmitrii Stepanov	6016d78a45	[#223] core: Refactor object format validator ... Resolve funlen linter for FormatValidator.ValidateContent method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-10 13:56:09 +03:00
Dmitrii Stepanov	02831d427b	[#213] metrics: Refactor object metrics ... Resolve funlen linter for newObjectServiceMetrics function. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-10 09:43:24 +03:00
Dmitrii Stepanov	38ae71cc7d	[#213] metrics: Refactor engine metrics ... Resolve funlen linter for newEngineMetrics function. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-10 09:43:24 +03:00
Evgenii Stratonikov	0b9622c418	[#191] control: Add Doctor RPC ... Doctor RPC performs complex operations on the storage engine. Currently only duplicate removal is supported. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-07 17:25:50 +00:00
Evgenii Stratonikov	dbc3811ff4	[#191] engine: Allow to remove redundant object copies ... RemoveDuplicates() removes all duplicate object copies stored on multiple shards. All shards are processed and the command tries to leave a copy on the best shard according to HRW. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-07 17:25:50 +00:00
Airat Arifullin	cb172e73a6	[#228] node: Use uber atomic package instead standard ... Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-07 15:37:27 +00:00
Dmitrii Stepanov	c236b54a65	[#212] reputationsvc: Resolve funlen linter ... Resolve funlen linter for Calculator.iterateDaughter method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 15:35:57 +00:00
Dmitrii Stepanov	7ebbfa3358	[#212] reputationsvc: Resolve linters and rename ... Resolved containedctx linters. Renamed context structs and interfaces to more understandble names. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 15:35:57 +00:00
Dmitrii Stepanov	469e8a6e59	[#212] reputationsvc: Resolve containedctx linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 15:35:57 +00:00
Dmitrii Stepanov	e2f13d03d7	[#222] auditsvc: Refactor PoR audit ... Resolve funlen linter for Context.checkStorageGroupPoR method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 17:29:13 +03:00
Dmitrii Stepanov	e8d340287f	[#222] auditsvc: Refactor audit task ... Resolve containedctx linter. Cancel task by listen cancel. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 17:29:13 +03:00
Dmitrii Stepanov	3dbff0a478	[#222] auditsvc: Resolve containedctx linter ... Resolve containedctx linter for commonCommunicatorPrm type. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 17:29:13 +03:00
Dmitrii Stepanov	fe87735073	[#219] morph: Refactor notary preparator ... Resolve funlen linter for Preparator.Prepare method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 15:43:20 +03:00
Dmitrii Stepanov	d07e40d6fe	[#219] morph: Refactor moprh event listener ... Resolve funlen and gocognit linters for listener.listenLoop method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 15:43:20 +03:00
Dmitrii Stepanov	775179f823	[#219] morph: Refactor notary invoke ... Resolve funlen linter for Client.notaryInvoke method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 15:43:20 +03:00
Dmitrii Stepanov	e815b19101	[#219] morph: Resolve containedctx linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-07 15:43:20 +03:00
Airat Arifullin	56282edf02	[#166] node: Parallelize background tree service sync ... * Run sync task for nodes in parallel within errgroup worker pool Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-07 04:43:32 +00:00
Dmitrii Stepanov	4ec69cbbf8	[#203] control: Fix test ... Fix shard compare loop. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Dmitrii Stepanov	f32f61df87	[#203] pilorama: Refactor tests ... Do not pass 0 as channel capacity. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Dmitrii Stepanov	8908798f59	[#203] node: Resolve unused vars ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Dmitrii Stepanov	bab11492ad	[#203] node: Resolve never used errors ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Dmitrii Stepanov	68a2f36636	[#203] morph: Mark depracated methods ... Skip staticcheck for depracated methods. Will be fixed soon. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Dmitrii Stepanov	9e2df4b7c7	[#203] node: Fix double imports ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Airat Arifullin	ab891517de	[#116] node: Fix bug with extra generated files in TestReload ... * Create testNewEngine in engineWithShards without default opts Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-05 18:49:34 +03:00
Dmitrii Stepanov	c58ab0c369	[#193] getsvc: Reduce private key requests ... Get private key only once for request forwaring. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:48 +00:00
Dmitrii Stepanov	89924071cd	[#193] getsvc: Edit request forwarder signature ... Pass context to forwarder direct, without closure. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:48 +00:00
Dmitrii Stepanov	6c7b708a98	[#193] getsvc: Refactor get range params creation ... Resolve funlen linter for toRangePrm function. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:48 +00:00
Dmitrii Stepanov	b0786d2e5c	[#193] getsvc: Refactor get params creation ... Resolve funlen linter for toPrm function. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:48 +00:00
Dmitrii Stepanov	f889893216	[#193] getsvc: Refactor head param creation ... Resolve funlen linter for toHeadPrm method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:48 +00:00
Dmitrii Stepanov	91ead04fa4	[#193] getsvc: Resolve funlen linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:48 +00:00
Dmitrii Stepanov	1bf21dbb47	[#193] getsvc: Resolve context linters ... Resolve containedctx and contextcheck linters. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:48 +00:00
Dmitrii Stepanov	206458c841	[#217] containersvc: Resolve containedctx linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:00 +00:00
Dmitrii Stepanov	279261ace3	[#217] containersvc: Refactor route passing ... Resolve containedctx for routeCtx. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:38:00 +00:00
Airat Arifullin	6f7b6a8813	[#116] node: Improve shard/engine construction in tests ... * Introduce testEngineWrapper that can be constructed with different options Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-05 14:36:40 +00:00
Dmitrii Stepanov	d6486d172e	[#210] policier: Refactor nodes processing ... Resolve funlen linter for processNodes method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:55:52 +03:00
Dmitrii Stepanov	080be5cfcd	[#210] policier: Refactor object placement ... Resolve containedctx and contextcheck linters. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:55:52 +03:00
Dmitrii Stepanov	23575e1ac0	[#210] policier: Resolve contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:55:52 +03:00
Evgenii Stratonikov	9098d0eec0	[#211] engine: Unify shard mode checks for tree operations ... All operations must ensure the shard is not in a degraded mode. Write operations must also ensure the shard is not in a read-only mode. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-05 11:10:39 +00:00
Evgenii Stratonikov	760af6b912	[#211] fstree: Consider ENOSPC a logical error ... We already do this for file writing, however directory creation can also fail. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-05 11:10:39 +00:00
Dmitrii Stepanov	d85703a963	[#208] searchsvc: Refactor request forwarding ... Resolve funlen & gocognit linters for toPrm method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 16:24:55 +03:00
Dmitrii Stepanov	0b38419fbf	[#208] searchsvc: Resolve context linters ... Resolve containedctx and contextcheck linters. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 16:24:50 +03:00
Dmitrii Stepanov	5f2a1531fe	[#208] deletesvc: Resolve containedctx linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 16:24:40 +03:00
Dmitrii Stepanov	4941926c9d	[#207] aclsvc: Drop outdated tag ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 13:22:14 +00:00
Dmitrii Stepanov	585415fa92	[#207] aclsvc: Refactor send checker ... Resolve funlen linter for putStreamBasicChecker.Send method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 13:22:14 +00:00
Dmitrii Stepanov	9ef790f782	[#207] aclsvc: Refactor object headers read ... Resolve funlen linter for readObjectHeaders method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 13:22:13 +00:00
Dmitrii Stepanov	cd33a57f44	[#207] aclsvc: Refactor EACL check ... Resolve funlen linter for CheckEACL method. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 13:22:13 +00:00
Dmitrii Stepanov	1f1aed87be	[#188] metabase: Refactor object inhume ... Resolve funlen linter for db.Inhume method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	5a66db80c5	[#188] engine: Refactor shard evacuation ... Resolve funlen and gocognit linter for StorageEngine.Evacuate method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	456bc097f7	[#188] engine: Refactor get range from engine ... Resolve funlen linter for StorageEngine.getRange method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	3010ca2649	[#188] engine: Refactor get object from engine ... Resolve funlen linter for StorageEngine.get method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	0739c36a3b	[#188] metabase: Refactor object put to metabase ... Resolve funlen linter for db.put method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	8273a3dfb2	[#188] blobstor: Refactor blobstor test ... Resolve funlen linter for TestIterate function Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	594b5821ed	[#188] blobstor: Refactor put data to blobovniczas ... Resolve funlen linter for Blobovniczas.Put method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Evgenii Stratonikov	ee7468daa7	[#205] innerring: Provide alphabetState param to epochTimer ... Fix NPE, introduced in f09ee27a. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-04 10:47:31 +00:00
Dmitrii Stepanov	8e5a0dcf27	[#204] gc: Fix GC handlers start ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 06:48:27 +00:00
Evgenii Stratonikov	0948a280fa	[#195] morph: use blocking unlimited pool for notifications ... With non-blocking pool restricted by 10 in capacity, the probability of dropping events is unexpectedly big. Notifications are an essential part of the FrostFS, we should not drop anything, especially new epochs. ``` Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z debug subscriber/subscriber.go:154 new notification event from sidechain {"name": "NewEpoch"} Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z warn event/listener.go:248 listener worker pool drained {"chain": "morph", "capacity": 10} ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-03 15:58:16 +00:00
Dmitrii Stepanov	ece6c820e7	[#199] putsvc: Refactor streamer initialization ... Resolve funlen linter for initTarget method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-03 15:58:11 +00:00
Dmitrii Stepanov	27bdddc48f	[#199] putsvc: Refactor put object ... Resolve containedctx linter for streamer and remote target Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-03 15:58:11 +00:00
Dmitrii Stepanov	cecea8053a	[#199] putsvc: Refactor streamer pool ... Resolve staticcheck linter for putBytesPool Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-03 15:58:11 +00:00
Dmitrii Stepanov	14d894178e	[#199] putsvc: Refactor placement iterator ... Resolve funlen linter for iteratePlacement method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-03 15:58:11 +00:00
Pavel Karpy	a69c6d1ec9	[#2272] morph: Do not subscribe to events without listening ... It led to a neo-go dead-lock in the `subscriber` component. Subscribing to notifications is the same RPC as any others, so it could also be blocked forever if no async listening (reading the notification channel) routine exists. If a number of subscriptions is big enough (or a caller is lucky enough) subscribing loop might have not finished subscribing before the first notification is received and then: subscribing RPC is blocked by received notification (non)handling and listening notifications routine is blocked by not finished subscription loop. That commit starts listening notification channel _before_ any subscription actions. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-03 08:42:41 +00:00
Pavel Karpy	aa92f977ef	[#181] ir: Do not pay for audit by non-alphabet nodes ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-31 12:53:37 +03:00
Pavel Karpy	f09ee27af9	[#181] ir: Do not process container estimations by non-alphabet nodes ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-31 12:53:09 +03:00
Pavel Karpy	db5321309d	[#181] ir: Do not sync the Alphabet by non-alphabet nodes ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-31 12:52:16 +03:00
Pavel Karpy	44d5412e10	[#181] ir: Do not deposit notary GAS by non-alphabet nodes ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-31 12:51:12 +03:00
Dmitrii Stepanov	56161d39b4	[#168] node: Refactor container services ... Resolve containedctx for remoteLoadAnnounceWriter struct Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-31 09:32:59 +03:00
Dmitrii Stepanov	a7c79c773a	[#168] node: Refactor node config ... Resolve containedctx linter for cfg Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-31 09:32:59 +03:00
Dmitrii Stepanov	8426d25f4b	[#185] ir: Resolve containedctx linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 19:10:33 +00:00
Dmitrii Stepanov	c1cbe6ff2d	[#185] ir: Refactor signature verification ... Resolve funlen linter for verifySignature method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 19:10:33 +00:00
Dmitrii Stepanov	aeb4bbc51e	[#185] ir: Refactor alphabet update ... Resolve funlen linter for processAlphabetSync method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 19:10:33 +00:00
Dmitrii Stepanov	30c18d46cc	[#185] ir: Refactor audit ... Resolve funlen linter for processStartAudit method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 19:10:33 +00:00
Dmitrii Stepanov	6be2688fb4	[#185] ir: Refactor gas emit ... Resolve funlen linter for processEmit method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 19:10:33 +00:00
Dmitrii Stepanov	4d160bd4ab	[#185] ir: Refactor ir service creation ... Resolve funlen linter for New function Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 19:10:33 +00:00
Dmitrii Stepanov	c8a6978563	[#185] ir: Refactor ir start ... Resolve funlen linter for Server.Start method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 19:10:33 +00:00
Dmitrii Stepanov	85fb9e77c4	[#185] ir: Resolve funlen linter ... Resolve funlen linter for GlagoliticLetter.String method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 19:10:33 +00:00
Airat Arifullin	7be5a0fd79	[#189] node: Fix bug with sync/atomic package ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2023-03-30 14:50:43 +00:00
Dmitrii Stepanov	ab32067152	[#183] gc: Fix drop expired locked complex objects ... Do not delete bucket keys during iteration Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 15:33:42 +03:00
Dmitrii Stepanov	9f0bce5c15	[#183] gc: Fix drop expired locked simple objects ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 15:33:42 +03:00
Alejandro Lopez	341fe1688f	[#139] test: Add test storage implementation ... This aims to reduce the usage of chmod hackery to induce or simulate OS-related failures. Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-03-29 14:28:49 +00:00
Pavel Karpy	e843e7f090	[#184] ir: Delete unused config reader param ... Not used since notary environments do endless and non-configurable deposits. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-29 10:34:54 +00:00
Pavel Karpy	ba58a77f8c	[#184] node, ir: Do not wait for already made notary deposits ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-29 10:34:54 +00:00
Pavel Karpy	3646723ae3	[#184] morph: Make deposit logs INFO ... We have already had and solved plenty of deposit issues and notary balance is a really important thing. Deserves to be INFO even before the huge logs severity refactor, happens on an app start only. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-29 10:34:54 +00:00
Pavel Karpy	97e201993b	[#184] *: Unify done contexts handling ... If `ctx.Done()`, return `ctx.Err()` in every function that returns an error. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-29 10:34:54 +00:00
Airat Arifullin	221203beeb	[#180] node: Refactor panics in unit test ... * Replace panics in unit tests by require.NoError and t.Fatalf Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2023-03-29 12:39:07 +03:00
Dmitrii Stepanov	91717d4b98	[#176] morph: Resolve funlen linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-27 18:49:55 +03:00
Evgenii Stratonikov	382ecae96a	[#172] Use ContainersOf() for container list fetching ... Previously we were limited by ~2048 containers because of neo-go VM limits. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-27 14:32:55 +00:00
Evgenii Stratonikov	9e54646248	[#172] client/container: Support listing containers with containersOf ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-27 14:32:55 +00:00
Evgenii Stratonikov	bf7d80f44b	[#172] morph/client: Support iterators via neo-go session API ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-27 14:32:55 +00:00
Pavel Karpy	f3ff9fd251	[#73] morph: Rename vars that collide with package names ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-24 09:42:30 +00:00
Pavel Karpy	533e9f8b75	[#59] morph: Adopt updated neo-go client API for subs ... It does not use deprecated methods anymore but also adds more code that removes. Future refactor that will affect more components will optimize usage of the updated API. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-24 09:42:30 +00:00
aarifullin	34329d67ff	[#86] node: Fix unit test and linter errors ... Signed-off-by: Airat Arifullin <aarifullin@yadro.com>	2023-03-23 12:42:58 +03:00
Ayrat Arifullin	9808dec591	[#86] node: Move testing utils to one package ... Move testing utils from tests in local_object_storage package to unified testutil package Signed-off-by: Airat Arifullin <aarifullin@yadro.com>	2023-03-23 08:19:15 +00:00
Denis Kirillov	342e571d89	[#159] Add handle __SYSTEM__ sys attributes ... Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>	2023-03-22 17:35:20 +03:00
Pavel Karpy	da8da1c63a	[#98] fstree: Do not fail iteration over just removed files ... A directory is read and files are saved to a local variable. The iteration over such files may lead to a non-existing files reading due to a normal SN operation cycle and, therefore, may lead to a returning the OS error to a caller. Skip just removed (or lost) files as the golang std library does in similar situations: 5f1a0320b9/src/os/dir_unix.go (L128-L133). Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-22 07:26:27 +00:00
Dmitrii Stepanov	49234b915e	[#155] search-service: Fix search with ST ... Search should return only objects allowed in static session Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-22 07:21:09 +00:00
Dmitrii Stepanov	1637a3edce	[#155] search-service: Add search with ST test ... In case of session token (ST) with object IDs search should return only objects allowed in static session Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-22 07:21:09 +00:00
Evgenii Stratonikov	cbc2efb1d6	[#156] object/get: Make toHeadPrm() pass gocognit ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Evgenii Stratonikov	8014fdb21a	[#156] metabase: Make freePotentialLocks() pass linter checks ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Evgenii Stratonikov	fb13902db9	[#156] shard: Make refillMetabase() pass linter checks ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Evgenii Stratonikov	3f6b962349	[#156] services/tree: Pass context to replicationWorker() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Evgenii Stratonikov	5368c4207a	[#156] services/tree: Split syncLoop() in functions ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Evgenii Stratonikov	47e8c5bf23	[#156] pilorama: Remove CIDDescriptor from TreeApply() ... Initially it was there to check whether an update is being initiated by a proper node. It is now obsolete for 2 reasons: 1. Background synchronization fetches all operations from a single node. 2. There are a lot more problems with trust in the tree service, it is only used in controlled environments. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Pavel Karpy	9a4f40626c	[#128] IR: Do not try to emit GAS to nobody ... Fix sending GAS to an empty extra wallets receivers list. Also, send GAS to extra wallets even if netmap is empty. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-21 08:59:46 +00:00
Dmitrii Stepanov	5059dcc19d	[#145] shard-gc: Delete expired objects after locks ... GC deletes expired locks and objects sequentially. Expired locks and objects are now being deleted concurrently in batches. Added a config parameter that controls the number of concurrent workers and batch size. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 11:31:08 +03:00
Dmitrii Stepanov	6c4a1699ef	[#145] shard-gc: Expired locked unit test ... Added unit test that verifies that GC deletes expired locked objects in one epoch. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 11:31:08 +03:00
Pavel Karpy	9cd8f7cea0	[#152] IR: Process empty basic incomes ... If network is not configured for basic income earnings, do not distribute GAS by the Alphabet nodes. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-21 06:59:33 +00:00
Dmitrii Stepanov	44b86bac5a	[#148] linter: Add contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 09:54:41 +03:00
Dmitrii Stepanov	481a1ca6f3	[#148] linter: Add gocognit linter ... Code with high cognitive complexity is hard intuitively to understand Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 09:54:41 +03:00
Dmitrii Stepanov	97c36ed3ec	[#148] linter: Add funlen linter ... Long functions are hard to understand and source of errors Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 09:54:41 +03:00
Dmitrii Stepanov	cc8ff015b4	[#148] linter: Add containedctx linter ... Context has to be passed as an argument: https://pkg.go.dev/context Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 09:52:39 +03:00
Dmitrii Stepanov	2dc86058c3	[#148] memstore: Drop space line ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 09:52:39 +03:00
Artem Tataurov	db3ccd2762	[#128] innerring: Add GAS pouring mechanism for a configurable list of wallets ... Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>	2023-03-20 12:50:05 +00:00
Pavel Karpy	abd21f8099	[#136] Revert "[#2260] services/object: Do not assemble object with TTL=1" ... This reverts commit 2567f8020e. It assumes that assembling logic could break some failover scenarios if request forwarding is done. However, it also breaks requesting big objects via a non-container node with TTL=2. Failover has been rechecked without that commit and no problems were found. Any (if found) other bugs related to the forwarding and object assembling must be solved more carefully. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-20 10:52:18 +03:00
Pavel Karpy	64bde68fb9	[#67] node: Accept expired locked objects ... Allow replication of any (expired too) locked object. Information about object locking is considered to be presented on the _container nodes_. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-16 16:22:19 +03:00
Pavel Karpy	f006f3b342	[#67] node: Make engine's IsLocked public ... It will allow reusing that method in expiration checks. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-16 16:20:45 +03:00
Evgenii Stratonikov	22be532cbd	object/put: Persist session token till the end of a session ... Previously a token could've expired in the middle of an object.PUT stream, leading to upload being interrupted. This is bad, because user doesn't always now what is the right values for the session token lifetime. More than that, setting it to a very high value will eventually blow up the session token database. In this commit we read the session token once and reuse it for the whole stream duration. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-16 06:45:50 +00:00
Alejandro Lopez	724debfdcd	[#81] node: Add basic read/write benchmarks for substorages ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-03-15 16:37:04 +00:00
Dmitrii Stepanov	ac0a278a05	[#85] get-service: Drop unused assemble flag ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-15 09:19:45 +03:00
Dmitrii Stepanov	b8e93d4c08	[#85] get-service: Use assembler to assemble LOB ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-15 09:19:45 +03:00
Dmitrii Stepanov	07de839f18	[#85] get-service: Fix corrupted chain logic ... Should return an error in case of a broken LOB reference chain. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-15 09:19:45 +03:00
Dmitrii Stepanov	2886b1581b	[#85] get-service: Add unit tests ... Add unit tests to cover all assemble statements Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-15 09:19:45 +03:00
Dmitrii Stepanov	8b9e40a848	[#85] get-service: Add assembler ... Extract assemble logic to assembler Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-15 09:19:45 +03:00
Evgenii Stratonikov	6cd806f998	[#82] services/tree: Save last synchronized height in a persistent storage ... Remember the last synchronized height and use it after service restart. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-13 11:25:44 +00:00
Evgenii Stratonikov	3e6fd4c611	[#82] pilorama: Allow to store last sync height ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-13 11:25:44 +00:00
Anton Nikiforov	5ae4446280	[#50] ir: Add Health status ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-03-13 11:24:42 +00:00
Anton Nikiforov	5890cd4d7d	[#50] ir: Fix config property name for prometheus ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-03-13 11:24:42 +00:00
Evgenii Stratonikov	05471d3827	[#83] util/autocomplete: Fix deprecated warning ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-13 07:07:30 +00:00
Evgenii Stratonikov	861e9ab59a	[#83] pre-commit: Add initial configuration ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-13 07:07:29 +00:00
Evgenii Stratonikov	6226c3ba86	[#129] policer: Use safer defaults ... If `processNodes` exits earlier for some reason, `needLocalCopy` could be false. See https://github.com/nspcc-dev/neofs-node/issues/2267 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-10 10:59:15 +00:00
Evgenii Stratonikov	f2250a316f	[#129] tree: Do not remove tree if the netmap is empty ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-10 10:59:15 +00:00
Pavel Karpy	f1f3c80dbf	[#32] node: Init write-cache asynchronously ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-09 11:07:33 +00:00
Pavel Karpy	381e363a8b	[#32] node: Always close general components after testing ... It will prevent test fails with `-race` flag on components that have background processes and make some actions on test framework. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-09 11:07:33 +00:00
Alex Vanin	20de74a505	Rename package name ... Due to source code relocation from GitHub. Signed-off-by: Alex Vanin <a.vanin@yadro.com>	2023-03-07 16:38:26 +03:00
Anton Nikiforov	e9f3c24229	[#65] Use strings.Cut instead of strings.Split* where possible ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-02-28 13:39:14 +03:00
Dmitrii Stepanov	6925fb4c59	[TrueCloudLab/hrw#2] node: Use typed HRW methods ... Update HRW lib and use typed HRW methods to sort shards and nodes Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-02-28 13:36:25 +03:00
Dmitrii Stepanov	c3a7039801	[TrueCloudLab/hrw#2] node: Optimize shard hash ... Compute shard hash only once Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-02-28 13:36:25 +03:00
Alejandro Lopez	73bb590cb1	[#64] node: Use pool_size_local and separate pool for local puts ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-02-22 13:43:19 +03:00
Alejandro Lopez	cb5468abb8	[#66] node: Replace interface{} with any ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-02-21 16:47:07 +03:00
Denis Kirillov	87e69b9349	[#44] node: Support multiple configs ... Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>	2023-02-21 10:00:28 +03:00
Pavel Karpy	337049b2ce	[#56] node: Allow reading expired locked object ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-21 09:56:57 +03:00
Pavel Karpy	3beef10f89	[#61] node: Do not fetch missing objects ... If an object is missing in a `meta`, shard should not look for it in a `blobstor`. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 14:47:38 +03:00
Anton Nikiforov	22f3c7d080	[#1868] Reload config for pprof and metrics on SIGHUP ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	0b61a3c961	[#2260] network/cache: Ignore clients only on Dial errors ... The problem is that accidental timeout errors can make us to ignore other nodes for some time. The primary purpose of the whole ignore mechanism is not to degrade in case of failover. For this case, closing connection and limiting the amount of dials is enough. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	bf1e59bb83	[#2260] network/cache: Ignore context cancelled errors ... Timeouts on client side should node affect inter-node communication. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	2567f8020e	[#2260] services/object: Do not assemble object with TTL=1 ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	d1d123d180	[#2234] writecache: Fix possible panic in initFlushMarks ... In case we have many small objects in the write-cache, `indices` should not be reused between iterations. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	315141dc2c	[#2252] fstree: Allow concurrent writes ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Pavel Karpy	b422ac9f94	[#2164] node: Fix multi-client error reporting ... Missing `ReportError` method did not allow casing multi-client interface to `errorReporter` interface and dropping broken connections. `replicationClient` embeds that interface, and it is widely used across node's code. Embedded interface does not allow casting its parent structure to `errorReporter` and breaks multi client error reporting logic. Multi-client scheme is extremely hard to maintain, it makes unpredictable casts and does not allow tracking code flow, so it will be refactored in the future anyway. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 13:53:27 +03:00
Pavel Karpy	95ee905861	[#2244] node: Fix subscriptions lock ... Subscribing without async listening could lead to a dead-lock in the `neo-go` client. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 13:53:27 +03:00
Pavel Karpy	07ec51ea60	[#2244] node: Add object address to WC's operations ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 13:53:27 +03:00
Pavel Karpy	dbbbef9ddb	[#2244] node: Update expired storage ID by WC ... Previously, node could get an "infinite" small object: it could be expired and thus could not be flushed (update its storage ID) to metabase => could not be marked as flushed => node never removes such object and repeat all the cycle one more time. If object exists and is not marked with GC (meta returns `ErrObjectIsExpired`, not `ObjectNotFound` and not `ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt transaction. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	6fd88a036f	[#2241] metrics: Fix request count metrics names ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	5cb2c5ae62	[#2238] engine: Add test for component initialization failures ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	427fe276f2	[#2238] shard: Try closing all components ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	c53903ccd0	[#2238] engine: Make Open and Init similar ... 1. Both could initialize shards in parallel. 2. Both should close shards after an error. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	e0309e398c	[#2239] writecache: Fix possible deadlock ... LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction. `View` transaction itself takes `mmapLock` [1], which is lifted after tx finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`) When we evict items from LRU cache mutex order is different: first we take LRU mutex and then execute `Batch` which _does_ take `mmapLock` in case we need to remap. Thus the deadlock. [1] 8f4a7e1f92/db.go (L708) Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	58367e4df6	[#2232] pilorama: Merge in-queue batches ... To achieve high performance we must choose proper values for both batch size and delay. For user operations we want to set low delay. However it would prevent tree synchronization operations to form big enough batches. For these operations, batching gives the most benefit not only in terms of on-CPU execution cost, but also by speeding up transaction persist (`fsync`). In this commit we try merging batches that are already _triggered_, but not yet _started to execute_. This way we can still query batches for execution after the provided delay while also allowing multiple formed batches to execute faster. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Pavel Karpy	40822adb51	[#2213] node: Do not return object expired object ... "Object is expired" means that object is presented in `meta` but it is not `ObjectNotFound` error. Previous implementation made `shard` search for an object without `meta` which was an error. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	9afe86ba3e	[#2212] morph: Fix subscription restoration ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Anton Nikiforov	85cf1f47ac	[#1465] node: Prevent process from killing by systemd when shutting down ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-02-17 12:13:00 +03:00
Artem Tataurov	362f24953a	[#47] shard: Switch container size metric from physical to logical capacity ... Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>	2023-02-17 12:03:42 +03:00
Pavel Karpy	901d62567d	[#57] node: Broadcast link objects ... It boosts object assembling by an _average_ container node. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-17 11:58:27 +03:00
Evgenii Stratonikov	204cd3a11c	[#31] fstree: Optimize treePath ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-10 12:49:31 +03:00
Evgenii Stratonikov	dee4498c1e	[#31] fstree: Do not check for a file existence twice ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-10 12:49:31 +03:00
Evgenii Stratonikov	abbecf49d6	[#31] fstree: Speedup string-to-address conversion ... ``` name old time/op new time/op delta _addressFromString-8 1.25µs ±30% 1.02µs ± 6% -18.49% (p=0.000 n=9+9) name old alloc/op new alloc/op delta _addressFromString-8 352B ± 0% 256B ± 0% -27.27% (p=0.000 n=9+10) name old allocs/op new allocs/op delta _addressFromString-8 6.00 ± 0% 4.00 ± 0% -33.33% (p=0.000 n=10+10) ``` Also, assure compiler that `s` doesn't escape: Before this commit: ``` ./fstree.go:74:24: leaking param: s ./fstree.go:90:6: moved to heap: addr ``` After this commit: ``` ./fstree.go:74:24: s does not escape ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-10 12:49:31 +03:00
Artem Tataurov	ab21d90cfb	[#1794] shard: Add increasing case for the payload size metric ... Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>	2023-02-09 13:30:23 +03:00
Stanislav Bogatyrev	cb016d53a6	[#1] Fix comments and error messages ... Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>	2023-02-06 17:41:14 +03:00
Stanislav Bogatyrev	c761a95eef	[#1] Fix project name in control service ... Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>	2023-02-06 17:41:14 +03:00
Pavel Karpy	73bc1b0b68	[#38] node: Fix linter warnings ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-06 17:27:54 +03:00
Anton Nikiforov	515c60bdf4	[#1889] adm: Add command morph netmap-candidates ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-02-06 17:26:34 +03:00
Pavel Karpy	89a0266f5e	[#1794] metrics: Track physical object capacity per shard ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-01-26 20:06:28 +03:00
Evgenii Stratonikov	9513f163aa	[#2116] metrics: Track physical object capacity in the container ... Currently we track based on `PayloadSize`, because it is already stored in the metabase and it is easier to calculate without slowing down the whole system. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com> Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-01-26 20:06:28 +03:00
Evgenii Stratonikov	d65a95a2c6	[#28] pilorama: Remove LogMove struct ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	46c62be7e8	[#28] Fix linter issues ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	c72576e72f	[#2208] engine: Log time-consuming shard operations ... Currently the only way to tell whether `evacuate/set-mode` is finished is to set a very big timeout and _hope_ that the operation will finish. In this commit we add INFO logs for such operations which should simplify the life of an administrator. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	87f0e3ea25	[#2208] fstree: Rename file after write ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	792319a044	[#2208] fstree: Remove file if there was an error during write ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	67c97c6804	[#2210] services/tree: Drop messages not in queue ... Currently, under high load clients are blocked on channel send and the number of goroutines can increase indefinitely. In this commit we drop replication messages if send/recv queue is full and rely on a background synchronization. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	25d5995cef	[#2210] pilorama: Allocate bucket name outside of batches ... 1. Reduce allocations inside transactions. 2. Do not encode container ID to string: it allocates a lot and takes more space. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	165a600624	[#2210] pilorama: Reduce the amount of keys per node ... Under high load we are limited by the _amount_ of keys we need to update in a single transaction. In this commit we try storing all state with a single key. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Pavel Karpy	64a5294b27	[#2200] shard: Do not fetch big objects from blobovniczas ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Pavel Karpy	91757329ae	[#2200] shard: Fix blobstor obj fetching ... In the previous implementation any non-nil error that preceded object fetching from blobstor led to iterating over every storage (in other words, no storage ID information was taken into account). Now storage ID is skipped only if metabase (storage ID source) returns any error. Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Pavel Karpy	cf1a91a758	[#2206] blobovnicza: Use Latin letters in the code ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Pavel Karpy	c33ad3c474	[#2164] node: Use reconnect_interval from config ... Not always the default one. Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	0d8366f475	[#2207] object/acl: Return status error for expired session token ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	6451f019d2	[#2203] shard: Do not panic in Close after unsuccessful Init ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	6efa93be0a	[#1621] services/tree: Return Apply result asyncronously ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	ac81c70c09	[#1621] pilorama: Batch related operations ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	9009612a82	[#2198] blobovniczatree: Properly handle concurrent active blobovnicza update ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	cedbd380f2	[#2197] pilorama: Close database in degraded mode ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Pavel Karpy	1d21b1e3e8	[#1978] node: Do not drop clients on split errors ... After the reconnection interval feature there was an bug related to the big objects collecting: split error is returned from a client directly, not via API status and was considered as a connection error. Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	554b85411f	[#2190] services/object: Log service error with INFO level ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	b0ad1b9ed2	[#2193] pilorama: Use do in TreeMove ... It should be similar to a `TreeAddByPath`. `applyOperation` is used for `Apply` when the operation can be inserted in the middle of a log. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	ba393e3e91	[#2188] engine: Fix panic during setting shard mode ... Under load changing shard mode can lead to it being removed from the list during some other PUT. ``` Dec 28 07:01:26 az neofs-node[364505]: panic: runtime error: invalid memory address or nil pointer dereference Dec 28 07:01:26 az neofs-node[364505]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc9fbb1] Dec 28 07:01:26 az neofs-node[364505]: goroutine 11791912 [running]: Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).putToShard(0xc000435490, {0xc0003f7a28?, 0xc0001192c0?}, 0x2, {0x0, 0x> Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:91 +0x1b1 Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put.func1(0xc000435490?, {0xc0003f7a28?, 0xc0001192c0?}) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:71 +0x19c Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).iterateOverSortedShards(0x1?, {{0x62, 0x23, 0xfe, 0x60, 0x67, 0xd5, 0x> Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:225 +0xc8 Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put(0xc000435490, {0x1?}) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:66 +0x2a9 Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put.func1() Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:43 +0x2a Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).execIfNotBlocked(0x8?, 0x38?) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:147 +0xcf Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put(0xc4df775a80?, {0x0?}) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:42 +0x65 Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.Put(0xc06d928b80?, 0xc06b1b8dc8?) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:158 +0x19 Dec 28 07:01:26 az neofs-node[364505]: main.engineWithoutNotifications.Put({0x20301b?}, 0x20301b?) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Anton Nikiforov	3d57f4c961	[#2179] test: Fix test TestEvacuateNetwork/multiple_shards,_evacuate_many ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-01-24 13:37:49 +03:00
Evgenii Stratonikov	3d1d2ee7b1	[#11] Regenerate proto files ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-12 08:55:47 +03:00
Evgenii Stratonikov	054bc4a727	[#11] Rename contract-related NeoFS occurences ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-12 08:55:47 +03:00
Evgenii Stratonikov	9cb4b4cc17	[#11] Rename neofsid contract to frostfsid ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-12 08:55:47 +03:00
Evgenii Stratonikov	19d180b510	[#11] Rename neofs contract to frostfs ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-12 08:55:47 +03:00
Evgenii Stratonikov	9936b112b8	[#5] blobstor: Use generic LRU cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	f0be0befc5	[#5] services/object_manager: Use generic LRU cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	1b3374ac7f	[#5] services/tree: User generic LRU cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	d0a0432a51	[#5] innerring: Use generic LRU cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	4155c1bdff	[#5] writecache: Use generic LRU cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	8f61cc1dcc	[#5] policer: Use generic LRU client ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	55b403e0ee	[#5] morph/client: Use generic LRU cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	0272218eb9	[#2184] compression: Properly calculate upper bound ... If the data is not compressible allocating `len(data)` will lead to a slice reallocation. For a compressible data the results for small size are flaky and we allocate a bit more. However, it feels right to use a provided function if we need to pick any size at all. ``` name old time/op new time/op delta Compression/size=128/zeroed_slice-8 2.23µs ±12% 2.06µs ± 6% -7.35% (p=0.009 n=10+10) Compression/size=128/not_so_random_slice_(block_=_123)-8 19.0µs ±10% 15.8µs ±16% -17.09% (p=0.000 n=9+10) Compression/size=128/random_slice-8 17.6µs ±15% 16.1µs ±16% ~ (p=0.075 n=10+10) Compression/size=1024/zeroed_slice-8 3.05µs ±11% 2.84µs ±10% ~ (p=0.089 n=10+10) Compression/size=1024/not_so_random_slice_(block_=_123)-8 18.1µs ± 6% 18.2µs ±12% ~ (p=0.971 n=10+10) Compression/size=1024/random_slice-8 48.6µs ± 6% 45.6µs ± 5% -6.07% (p=0.006 n=10+9) Compression/size=32768/zeroed_slice-8 26.8µs ± 3% 28.7µs ± 8% +7.23% (p=0.001 n=10+10) Compression/size=32768/not_so_random_slice_(block_=_123)-8 44.3µs ± 8% 43.7µs ±13% ~ (p=0.762 n=8+10) Compression/size=32768/random_slice-8 97.3µs ±32% 68.9µs ±15% -29.13% (p=0.000 n=10+10) Compression/size=33554432/zeroed_slice-8 29.8ms ± 9% 30.3ms ±17% ~ (p=1.000 n=9+9) Compression/size=33554432/not_so_random_slice_(block_=_123)-8 33.1ms ±14% 30.3ms ±11% -8.61% (p=0.043 n=10+10) Compression/size=33554432/random_slice-8 41.7ms ± 3% 30.1ms ± 8% -27.72% (p=0.000 n=9+10) name old alloc/op new alloc/op delta Compression/size=128/zeroed_slice-8 128B ± 0% 144B ± 0% +12.50% (p=0.000 n=10+10) Compression/size=128/not_so_random_slice_(block_=_123)-8 384B ± 0% 144B ± 0% -62.50% (p=0.000 n=10+10) Compression/size=128/random_slice-8 384B ± 0% 144B ± 0% -62.50% (p=0.000 n=10+10) Compression/size=1024/zeroed_slice-8 1.02kB ± 0% 1.15kB ± 0% +12.50% (p=0.000 n=10+10) Compression/size=1024/not_so_random_slice_(block_=_123)-8 1.02kB ± 0% 1.15kB ± 0% +12.50% (p=0.000 n=10+10) Compression/size=1024/random_slice-8 2.56kB ± 0% 1.15kB ± 0% -55.00% (p=0.000 n=10+10) Compression/size=32768/zeroed_slice-8 32.8kB ± 0% 41.0kB ± 0% +25.00% (p=0.000 n=10+10) Compression/size=32768/not_so_random_slice_(block_=_123)-8 32.8kB ± 0% 41.0kB ± 0% +25.00% (p=0.000 n=10+10) Compression/size=32768/random_slice-8 81.9kB ± 0% 41.0kB ± 0% -50.00% (p=0.000 n=10+10) Compression/size=33554432/zeroed_slice-8 33.6MB ± 0% 33.6MB ± 0% +0.02% (p=0.000 n=9+9) Compression/size=33554432/not_so_random_slice_(block_=_123)-8 33.6MB ± 0% 33.6MB ± 0% +0.02% (p=0.000 n=8+10) Compression/size=33554432/random_slice-8 75.5MB ± 0% 33.6MB ± 0% -55.55% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Compression/size=128/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=128/not_so_random_slice_(block_=_123)-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) Compression/size=128/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) Compression/size=1024/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=1024/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=1024/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) Compression/size=32768/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=32768/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=32768/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) Compression/size=33554432/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=33554432/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=33554432/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	741482c26f	[#2164] neofs-node: Allow to set reconnect interval ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	6f5edac730	[#2164] network/cache: Do not reconnect to failed clients immediately ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	f3caf6acfe	[#2164] network/cache: Separate mutex for addr ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	0ace28e43d	[#2175] blobovniczatree: Close all non-active blobovniczas ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	c1cf418956	[#2175] blobovniczatree: Make function parameters more descriptive ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	b4e90cdf51	[#2165] pilorama: Optimize TreeApply when used for synchronization ... Because synchronization _most likely_ will have apply already existing operations, it is much faster to check their presence in a read transaction. However, always doing this will degrade the perfomance for normal `Apply`. And, let's be honest, it is already not good. Thus we add a separate parameter which specifies whether this logic is enabled. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	f9fcd85363	[#2165] services/tree: Remember starting height for the synchronization ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	06137dbf8e	[#2165] services/tree: Do not export synchronizeAllTrees ... It is used only privately. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	c299b98afe	[#2165] services/tree: Parallelize synchronization ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	de9957e076	[#2165] services/tree: Always synchronize all containers ... In case of split-brain we must synchronize everything. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	6a4e5e6f0a	[#2144] node: Try node's private key if dynamic token fetching failed ... `GETRANGEHASH` request spawns `GETRANGE` requests if an object could not be found locally. If the original request contains session, it can be static and, therefore, fetching session key can not be performed successfully. As the best effort a node could request object's range with its own key. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	86a4fba571	[#2144] node: Clarify KeyStorage.GetKey method ... Actualize the doc, fix API status error return. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	21717262ec	[#2016] shard: Check meta first on Get ... `meta` should prevent returning removed objects (`GCMark` and `TS` relations are `meta` abstractions). Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	74ec71446f	[#2167] shard: Do not use write-cache by default in Head ... Both `meta` and `write-cache` are expected to have a fast underlying disk, so it does not seem like an optimisation. Moreover, `write-cache`'s `Head` is a `Get` with payload cutting, it _must_ use more memory for no reason (`meta` was created for such requests). Also, `write-cache` does not allow performing any "meta" relations checks (such as locking, tombstoning). Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	1608fd1c07	[#2167] write-cache: Add "write-cache" to its logs ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	eea2892109	[#1956] node: Lock shard's mode on its methods switch ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	04b5ec759b	[#2139] object/put: Use sync.Pool for temporary payloads ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	9e0decd12d	[#2162] services/tree: Close connection after the syncronization ... There was a goroutine leak here. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	e1c3bdbfa6	[#1621] pilorama: Remove Timestamp field from nodeInfo ... It is already present in `Meta`. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	1044adbe94	[#1621] pilorama: Improve memory allocation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	2539d466a6	[#1621] pilorama: Seek after cursor invalidation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	e9ba8931f8	[#1621] pilorama: Simplify bucket creation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	fe7ddfdc6a	[#1621] pilorama: Compare memory forests properly ... Node children are not sorted and could occur in any order. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Pavel Karpy	306609030a	[#2159] node: Add tree replication timeout configuration ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Anton Nikiforov	edb1428248	[#2022] Add metric readonly to get shards mode ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	e5c304536b	[#2161] pilorama: Do not apply already existing operations ... Speeds up synchronization a bit. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	3bb5a320d7	[#2154] services/tree: Do not log an error when synchronizing container of 1 node ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	387d1e2977	[#2127] services/tree: Randomize node order for synchronization ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	21c58c92a9	[#2145] meta: Do allow force inhuming a locked object ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	b207dc424f	[#2158] policer: Reduce default cache size ... We use cache to avoid policing the same object multiple times in a short time span (< 30 seconds). If we have 200_000 objects in a blobstor, it is a bit useless -- if it takes 1 second to process an object and we have `replicator.pool_size: 20` in config, the next iteration will happen in 10_000 second which is much larger than 30 second. However we still consume a lot of memory, so it makes sense to use saner default. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	0244f2e5ce	[#2156] core/netmap: Change node addresses iteration order ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	b413094704	[#2095] node: Fix collecting child objects ... Stop child objects collection if the last returned object (the most "left" object in the collected chain) starts exactly from the `GETRANGE`'s `from` value. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-12-30 11:07:35 +03:00
Pavel Karpy	350eecfa13	[#2095] node: Do not allow GETRANGE requests with zero length ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	0b78af467e	[#2140] engine: Fix error handling in TreeMove ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	923f84722a	Move to frostfs-node ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-28 15:04:29 +03:00
Evgenii Stratonikov	42554a9298	[#2068] writecache: Remove deleted objects from the writecache ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Evgenii Stratonikov	4a49ea0855	[#2068] writecache: Allow to open FSTree in read-only mode ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Evgenii Stratonikov	857d2dc3f5	[#2068] writecache: Optimize initial flush existence checking ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Evgenii Stratonikov	63f604e948	[#2068] blobstor: Allow to provide storage ID in Exists ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Evgenii Stratonikov	6ad2b5d5b8	[#2068] blobovnicza: Add Exists method ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Alex Vanin	ba74244d1b	[#1486] ir: Use endless notary deposit for side chain ... Signed-off-by: Alex Vanin <a.vanin@yadro.com>	2022-12-02 11:43:33 +03:00
Alex Vanin	b6a40241f8	[#1486] morph: Add method for endless notary deposit ... Signed-off-by: Alex Vanin <a.vanin@yadro.com>	2022-12-02 11:43:33 +03:00
Pavel Karpy	d54022eacc	[#2047] node: Do not send chunk twice on request forwarding ... That could happen if a node forwards request to a node that closed the connection during the original object stream. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-12-02 11:27:48 +03:00
Evgenii Stratonikov	bd25db5d4a	[#1984] metrics: Use separate metrics for success/failed requests ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-01 14:18:10 +03:00
Evgenii Stratonikov	e21c472dc7	[#1984] services/object: Increase put_req_count after the request is processed ... As it is specified in metrics description. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-01 14:18:10 +03:00
Pavel Karpy	50d28b72c3	[#2075] node, ir: Log notary state with info ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 19:26:33 +03:00
Pavel Karpy	b8c30b88f6	[#2075] morph: Do not do notary deposit with zero balance ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 19:26:33 +03:00
Pavel Karpy	c01d4ecb50	[#2080] morph: Close morph clients ... Could be related to "websocket users limit reached" on the `neo-go` server side when an SN/IR is rebooting repeatedly. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 19:26:33 +03:00
Pavel Karpy	3d0768a1d3	[#2061] node: Unify meta.Get benchmarks ... Make them get exactly one (different) object per a bench iteration. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 18:29:14 +03:00
Pavel Karpy	bc905f169d	[#2061] meta: Add parallel bench for Get ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 18:29:14 +03:00
Pavel Karpy	960a2d0629	[#2081] ir: Set default key in IR's SDK clients ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 16:58:52 +03:00
Pavel Karpy	51963abce7	[#1972] node: Fix errors comments in the Put service ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 16:58:52 +03:00
Evgenii Stratonikov	7335a52f29	[#1732] pilorama: Improve logical error handling ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-30 16:58:52 +03:00
Evgenii Stratonikov	ae7b473768	[#2064] blobovniczatree: Remove index too big log ... There is no need to log about a situation which is expected. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-30 16:53:18 +03:00
Evgenii Stratonikov	c1ea6fd854	[#2091] morph/client: Use notary.FakeContractAccount wrapper ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-30 16:45:13 +03:00
Evgenii Stratonikov	b93be8869b	[#2091] morph/client: Simplify code ... 1. Replace `mn` function with a `sigCount`. 2. Use `notary.FakeMultisigAccount` for account creation. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-30 16:45:13 +03:00
Evgenii Stratonikov	6af52c46d8	[#2091] morph/client: Do not use deprecated methods ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-30 16:45:13 +03:00
Evgenii Stratonikov	816c74d185	[#2075] morph/client: Ignore error if a transaction already exists ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Vladimir Domnich	71853348b9	[#2063] morph/client: Support new hash format in morph nns client ... Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>	2022-11-19 11:01:04 +03:00
Pavel Karpy	ed4351aab0	[#2074] write-cache: Do not flush same object twice ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	dd225906a0	[#2074] write-cache: Remove unused variables ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	9f7ac6bf9f	[#2069] innerring: Do not panic in Head ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	660c38d07e	[#2062] services/policer: Use a proper key for object cache ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	b1025bdb42	[#2057] meta: Fail write operations in R/O mode ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	fdeea1dfac	[#2057] meta: Fix concurrent mode changes ... Includes: 1. mode change read lock operation in every exported method that r/w the underlying database; 2. returning `ErrDegradedMode` logical error if any exported method is called in degraded (without a metabase) mode. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	3d6defd3e8	[#2057] meta: Do not lock the whole meta on GET ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	fa231b8c56	[#2057] blobstor: Block operations on a mode change ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	1779664644	[#2058] services/policer: Fix panic in shardPolicyWorker ... ``` 2022/11/15 08:40:56 worker exits from a panic: runtime error: index out of range [0] with length 0 2022/11/15 08:40:56 worker exits from panic: goroutine 1188 [running]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1() github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c panic({0x1042b60, 0xc0015ae018}) runtime/panic.go:1038 +0x215 github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1() github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x366 github.com/panjf2000/ants/v2.(*goWorker).run.func1() github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97 created by github.com/panjf2000/ants/v2.(*goWorker).run github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x68 ``` Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	b673d9e472	[#2053] engine: Do not switch mode because of logical errors ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Anton Nikiforov	9a20498f34	[#1940] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2022-11-19 11:01:04 +03:00
Pavel Karpy	634792077e	[#1502] node: Store lock object on every container node ... Includes extending listing methods in the Storage Engine with object types. It allows tuning replication/policer algorithms: container nodes do not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement on the ohter container nodes. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	fe09cd9c70	[#1502] core: Add AddressWithType ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	3b61cb4f49	[#1502] engine: Check all shards for LOCK'ing before inhuming ... It allows keeping all the locked objects safe after metabase resynchronization. Currently, all `LOCK` objects are broadcast to all nodes in a container, it guarantees `LOCK` object presence in a regular situation. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	34e8d2ba56	[#1502] shard: Add IsLocked method ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	9a039ba582	[#1502] meta: Add IsLocked method ... It gets an object and returns its locking status. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	659011c143	[#2048] network/cache: Optimize ClientCache ... 1. Remove a layer of indirection for mutex, `ClientCache` is already used by pointer. 2. Fix duplication of a `AllowExternal` field. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	d5a14041e0	[#2040] node: Do not attach tokens in the assembly process ... A container node is expected to have full "get" access to assemble the object. A non-container node is expected to forward any request to a container node. Any token is expected to be issued for an original request sender not for a node so any new request is invalid by design with that token. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	fd61bdadcb	[#2040] node: Attach original meta to the spawned requests ... Do not lose meta information of the original requests: cache session and bearer tokens of the original request b/w a new generated ones. Middle request wrappers should not contain any meta information, since it is useless (e.g. ACL service checks only the original tokens). Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	d65604ad30	[#1985] blobstor: Allow to report multiple errors to caller ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	f2d7e65e39	[#2035] engine: Allow moving to degraded from background workers ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	481b48b942	[#2028] node: Check session token's NBF and IAT ... ACL service did not check "Not Valid Before" and "Issued At" claims. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	aadd2ad050	[#2028] node: Do not wrap malformed request errors ... After presenting request statuses on the API level, all the errors are unwrapped before sending to the caller side. It led to a losing invalid request's context. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	2522d924b9	[#2037] services/object: Fix concurrent map writes in traverser ... ``` fatal error: concurrent map writes goroutine 4337 [running]: github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*traversal).submitProcessed(...) github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:78 github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*distributedTarget).iteratePlacement.func1() github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:198 +0x265 github.com/panjf2000/ants/v2.(*goWorker).run.func1() github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97 created by github.com/panjf2000/ants/v2.(*goWorker).run github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65 ``` Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	b0e94b6a6b	[#1906] writecache: Do not require read-only mode in Flush ... It was needed before we started to flush during transition to `degraded` mode. Now it is confusing. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	aa478f1def	[#2024] services/object: Unify status errors ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	3875fef542	[#2024] services/object: Cover corner cases for children OutOfRange ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	b9fb0d6050	[#1972] node: Fix object format unit tests ... Includes: 1. Unused func removal; 2. Err check of the `Sign` method. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	aab398f4f5	[#1972] node: Do not save objects if node not in a container ... Do not use node's local storage if it is clear that an object will be removed anyway as a redundant. It requires moving the changing local storage logic from the validation step to the local target implementation. It allows performing any relations checks (e.g. object locking) only if a node is considered as a valid container member and is expected to store (stored previously) all the helper objects (e.g. `LOCK`, `TOMBSTONE`, etc). Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	2849e465f9	[#1699] meta: Do not return SplitInfoError on Delete ... It is not an error: removing virtual object is expected and should be just skipped. Getting a virtual object with `raw` flag is considered as an impossible action, all the virtual objects removals will be handled via their children's removals implicitly. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	a3e7365cbd	[#1732] pilorama: Fill parent mark correctly ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	134f2ba02e	[#1732] pilorama: Fix backwards log insertion ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	2ef38cfbc4	[#1996] engine: Ignore pilorama.ErrTreeNotFound for write operations ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	a455ec18c3	[#2007] services/object: Allocate memory on-demand in GET_RANGE ... For big objects we want to get OutOfRange error before all the memory is allocated. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	ff5526038d	[#2007] services/object: Fix comment ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	d8d3588e1b	[#1996] engine: Always select proper shard for a tree ... Currently there is a possibility for modifying operations to fail because of I/O errors and a new tree to be created on another shard. This commit adds existence check for modifying operations. Read operations remain as they are, not to slow things. `TreeDrop` is an exception, because this is a tree removal and trying multiple shards is not an unwanted behaviour. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-03 15:29:23 +03:00
Evgenii Stratonikov	777fd32d4f	[#1818] writecache: Increase error counter on background errors ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	bffb0f894c	[#1818] writecache: Update storage ID during flush ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	5cf75404dc	[#1818] metabase: Add UpdateStorageID operation ... By default writecache puts the whole object to update storage ID. This logic comes from the times when we needed to put objects in the metabase by the writecache itself. Now this is done by the blobstor at unmarshaling objects during flush only to update storage ID is an overkill. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	b64b14eb54	[#1818] writecache: Reuse FSTree flushing code between flushes ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	a56927e3d4	[#1818] writecache: Remove unused variable ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	98a152256b	[#1992] writecache: Allow to open in NOSYNC mode ... Applicable only to FSTree as we cannot handle corrupted databases properly yet. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-01 09:42:26 +03:00
Evgenii Stratonikov	f564430b90	[#1992] fstree: Allow working in SYNC mode ... Make O_SYNC the default and allow to opt-out explicitly. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-01 09:42:26 +03:00
Evgenii Stratonikov	34501685b7	[#1969] local_object_storage: Move ErrObjectIsExpired to another package ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-31 11:41:24 +03:00
Evgenii Stratonikov	56de2f1363	[#1969] local_object_storage: Simplify logic error construction ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-31 11:41:24 +03:00
Evgenii Stratonikov	fcdbf5e509	[#1969] local_object_storage: Add a type for logical errors ... All logic errors are wrapped in `logicerr.Logical` type and do not affect shard error counter. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-31 11:41:24 +03:00
Leonard Lyubich	e8c5f03c30	[#1905] shard: Don't log read-only errors of write-cache ... There is no need to log `writecache.ErrReadOnly` errors in `Delete` method of the `Shard`. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-28 18:30:45 +03:00
Leonard Lyubich	b1fa084756	[#1905] shard: Decrease severity level of write-cache failure logs ... In previous implementation `Shard.Delete` logged writecache's removal failures in `error` level. There is a need to decrease severity of these log records since they aren't critical and don't require individual review. Change level of the message to `info`. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-28 18:30:45 +03:00
Evgenii Stratonikov	846ff515e6	[#1812] policer: Do not remove copies if there are maintenance nodes ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 19:13:17 +03:00
Evgenii Stratonikov	1e6588e761	[#1974] shard: Do not panic in degraded mode ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 12:41:12 +03:00
Evgenii Stratonikov	7395ab8ef7	[#1944] metabase: Assume static buckets are created on Init ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 12:41:12 +03:00
Evgenii Stratonikov	e4bc3d0e9d	[#1944] metabase: Recreate static buckets instead of resetting ... From the `Bucket.ForEach` doc: ``` The provided function must not modify the bucket; this will result in undefined behavior. ``` Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 12:41:12 +03:00
Evgenii Stratonikov	3b939d190c	[#1957] engine: Move shard to read-only if cannot move to degraded ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 08:20:53 +03:00
Pavel Karpy	9a0824766c	[#1713] node: Add Neo RPC switch interval to the config ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-25 17:38:48 +03:00
Pavel Karpy	0631b38da1	[#1713] ir: Add Neo RPC switch interval to the config ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-25 17:38:48 +03:00
Pavel Karpy	d240e2dc87	[#1713] morph: Allow switch interval configuration ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-25 17:38:48 +03:00
Pavel Karpy	17f7d0a2ee	[#1615] morph: Switch to a more prioritized RPC node ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-25 17:38:48 +03:00
Pavel Karpy	7c0aa69d11	[#1615] morph: Fix the switch comment message ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-25 17:38:48 +03:00
Evgenii Stratonikov	713fdab177	[#1907] shard: Return from Close after GC has stopped ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-25 11:54:45 +03:00
Pavel Karpy	41f93dcc1d	[#1953] blobstor: Log any object removal operation ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-24 21:52:18 +03:00
Pavel Karpy	942cc38146	[#1953] engine: Add refactored storage helpers ... After the refactor there are new storage characteristics: a type and a general storage id (that could be stringified). Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-24 21:52:18 +03:00
Pavel Karpy	0371d15b2f	[#1953] shard: Fix debug log messages ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-24 21:52:18 +03:00
Pavel Karpy	f8180447a1	[#1938] meta: Make version error messages more descriptive ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-24 21:46:18 +03:00
Evgenii Stratonikov	926830bb9c	[#1961] network/cache: Do not return nil client ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 17:56:34 +03:00
Leonard Lyubich	7b418c36b4	[#1930] services/session: Log calling Create RPC ... There is a need to check if session is opened during system testing/debug. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-24 17:45:22 +03:00
Evgenii Stratonikov	8796807040	[#1936] network/cache: Handle dial errors ... After an SDK update `Dial` can return an error. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:53 +03:00
Evgenii Stratonikov	1beafea0b5	[#1869] shard: Add logs for SetMode operations on reload ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:26 +03:00
Evgenii Stratonikov	87be4f1629	[#1869] shard: Restore shard mode on failed reloads ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:26 +03:00
Evgenii Stratonikov	c785e11b20	[#1869] shard: Allow to reload metabase on SIGHUP ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:26 +03:00
Evgenii Stratonikov	f769fc83fc	[#1869] shard: Embed gcCfg as raw struct ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:26 +03:00
Leonard Lyubich	60e9de8d63	[#1916] control: Check maintenance allowance on Control server ... In previous implementation turning to maintenance mode using NeoFS CLI required NeoFS API endpoint. This was not convenient from the user perspective. It's worth to move networks settings' check to the server side. Add `force_maintenance` field to `SetNetmapStatusRequest.Body` message of Control API. Add `force` flag to `neofs-cli control set-status` command which sets corresponding field in the requests body if status is `maintenance`. Force flag is ignored for any other status. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-24 09:20:24 +04:00
Pavel Karpy	e1be0180f6	[#1329] tree: Sync trees when a node first time appears in a container ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-21 18:47:56 +03:00
Pavel Karpy	49c38d642d	[#1902] engine: Search for the tree IDs in every shard ... Iterate over every shard and search for the container's trees. Final result is a concatenation of shards' results. It is considered that one fixed tree is placed on one fixed shard but the different trees of a fixed container could be placed on different shards. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	1766ca2039	[#1902] tree: Allow synchronize all the container trees ... Add `SynchronizeAllTrees` method of the Tree service. It allows fetching tree IDs and sync all of them. Share common logic b/w the new method and the `SynchronizeTree`. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	6d4beea187	[#1902] tree: Extend grpc service with ListTrees method ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	24e9e3f3bf	[#1902] engine, shard: Implement TreeList method ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	2e199c7ab1	[#1902] shard: Fix pilorama disabled err message ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	19850ef157	[#1902] pilorama: Add TreeList method ... To both `bolt` and `memory` forests; extend `Forest` interface. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	3aa9938b8f	[#1902] Update protoc to v3.21.7 ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Evgenii Stratonikov	9ec01bb9c1	[#1931] control: Allow to clear errors in SetShardMode RPC ... It hasn't been working since the initial implementation 7fb15fa1d0. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-20 15:51:31 +03:00
Evgenii Stratonikov	0d65888005	[#1910] .golangci.yml: Add predeclared linker ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-18 15:08:26 +03:00
Evgenii Stratonikov	1cb892c579	[#1910] .golangci.yml: Add misspell linker ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-18 15:08:26 +03:00
Evgenii Stratonikov	d772e35aba	[#1910] .golangci.yml: Add godot linker ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-18 15:08:26 +03:00
Pavel Karpy	bc67f77d86	[#1329] tree: Fix sync applying ... Add the node position in a container and the container size to the CID descriptor that is passed to the `TreeApply`. Previously, `checkValid` does not allow any log operarations. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-17 20:53:34 +03:00
Pavel Karpy	f5f560d903	[#1329] tree: Do not sync trees that are from external containers ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-17 20:53:34 +03:00
Pavel Karpy	f76516a883	[#1860] meta: Fix 0,1 -> 2+ version migration ... In the 2nd version, there was a database format change: buckets have changed their keys, so it becomes impossible to check the version in the 1 -> 2+ migrations because of different buckets that store info about the version. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-17 13:42:05 +03:00
Leonard Lyubich	1406d096a2	[#1680] service/object: Fail all operations in maintenance mode ... Storage node should not provide NeoFS Object API service when it is under maintenance. Declare `Common` service that unifies behavior of all object operations. The implementation pre-checks if node is under maintenance and returns `apistatus.NodeUnderMaintenance` if so. Use `Common` service as a first logical processor in object service pipeline. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-14 13:54:32 +04:00
Leonard Lyubich	05420173cc	[#1894] services/object: Ignore unrelated sessions in client ... In some scenarios original session can be unrelated to the objects which are read internally by the node. For example, node requests child objects when removing the parent one. Tune internal NeoFS API client used by node's Object API server to ignore unrelated sessions in `GetObject` / `HeadObject` / `PayloadRange` ops. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-14 12:37:31 +03:00
Pavel Karpy	13c4a9f4b8	[#1332] tree: Make SignMessage public ... It will allow reusing signing routine in other components (e.g. `neofs-cli`). Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-13 20:01:48 +03:00
Pavel Karpy	31c623636d	[#1863] node: Fix shard id in the object counter metrics ... If shard ID is stored in metabase (it is not the first time boot), read it, set it, use it (not a generated one) in the metrics writer. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-13 13:06:41 +03:00
Anton Nikiforov	8714fc42b5	[#1765] Use hex format to print storage node ID ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2022-10-13 12:55:21 +03:00
Pavel Karpy	d7c7022bbd	[#1770] logger: Support runtime level reloading ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-12 18:11:05 +03:00
Pavel Karpy	f037022a7a	[#1770] logger: Refactor Logger component ... Make it store its internal `zap.Logger`'s level. Also, make all the components to accept internal `logger.Logger` instead of `zap.Logger`; it will simplify future refactor. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-12 18:11:05 +03:00
Evgenii Stratonikov	4baf00aa21	[#1884] services/object: Fallback to GET in GET_RANGE ... Current spec allows denying GET_RANGE requests from other storage nodes. However, GET should always be allowed and it is enough to perform GET_RANGE locally Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-12 17:05:51 +03:00
Leonard Lyubich	dde4d4df2a	[#1878] services/object: Fix child check in GET ... In previous implementation `ObjectService.Get` RPC handler failed with `parent address in child object differs` while assembling the "big" object. This was caused by the child check which required parent reference to be set in all child objects. The check was impracticable because not all elements of the split-chain have a link to the parent. Make `execCtx.isChild` to return `true` if parameterized object has no parent header in its own header. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-12 16:56:37 +03:00
Evgenii Stratonikov	49eab6318c	[#1867] control: Fix degraded-read-only mode parsing ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-12 11:20:48 +03:00
Evgenii Stratonikov	c0199dee93	[#1867] services/control: Interpret empty list of IDs as all shards ... In neofs-cli the flag is still required, but `all` can be used to process all shards. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-12 11:20:48 +03:00
Evgenii Stratonikov	19c0a74e94	[#1867] services/control: Allow to provide multiple shard IDs to some commands ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-12 11:20:48 +03:00
Anton Nikiforov	74d2f2c8d3	[#1854] cli: Unify help messages ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2022-10-12 09:53:11 +03:00
Leonard Lyubich	050ad2762c	[#1680] replicator: Consider NODE_UNDER_MAINTENANCE as OK ... Node response with `NODE_UNDER_MAINTENANCE` status signals that the node was switched to maintenance mode. There is a delay between the actual switch and the reflection in the network map of up to one epoch. To speed up the reaction to the maintenance, it is required to recognize such node responses in the Policer. Make `Policer.processNodes` to exclude elements with shortage decreasing on `NODE_UNDER_MAINTENANCE` status response. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-11 12:54:27 +03:00
Leonard Lyubich	e99e25b52f	[#1680] replicator: Consider nodes under maintenance as OK ... Nodes under maintenance SHOULD not respond to object requests. Based on this, storage node's Policer SHOULD consider such nodes as problem ones. However, to prevent spam with the new replicas, on the contrary, Policer should consider them normal. Make `Policer.processNodes` to exclude elements if `IsMaintenance()` with shortage decreasing. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-11 12:54:27 +03:00
Leonard Lyubich	df5d7bf729	[#1680] replicator: Work with netmap.NodeInfo in TaskResult ... Make `replicator.TaskResult` to accept `netmap.NodeInfo` type instead of uint64 in order to clarify the meaning and prevent passing the random numbers. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-11 12:54:27 +03:00
Leonard Lyubich	e6f8904040	[#1680] policer: Refactor tracking the processed nodes ... Add clear methods with docs. Use the methods instead of direct map and bool instructions. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-11 12:54:27 +03:00
Anton Nikiforov	feaa9eace7	[#1768] Add healthcheck method to the Tree service ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2022-10-11 12:53:45 +03:00
Leonard Lyubich	807c0a1321	[#1859] services/object: Do not session check relation in PUT ... It doesn't make sense to check object relation in session check of `ObjectService.Put` RPC which has been spawned by `ObjectService.Delete` with session. Session issuer can't predict identifier of the tombstone object to be created. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-10 20:09:47 +03:00
Evgenii Stratonikov	2d43892fc9	[#1840] neofs-node: Use blobstor paths to identify shard ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	4b005d3178	[#1840] blobstor: Return info about all components ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	6d7ffefec5	[#1840] blobstor/test: Add tests for auxiliary functions ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	328691c94f	[#1840] blobstor: Return Path from components ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	9b241e4a17	[#1840] neofs-node: Allow to use mode: disabled in config ... Currently, when removing shard special care must be taken with respect to shard numbering. `mode: disabled` allows to leave shard configuration in place while also ignoring it during initialization. This makes disk replacement much more convenient. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	b2aa9947c2	[#1829] engine: Delete split objects properly ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-07 16:35:46 +03:00
Evgenii Stratonikov	6557f5d249	[#1839] engine: Handle Inhume errors properly ... If shard is in read-only or degraded mode, there is no need to increase error counter. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-07 14:25:52 +03:00
Evgenii Stratonikov	7356ee91ff	[#1837] services/object: Optimize uniqueIDWriter ... Avoid encoding object ID to string. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-07 13:29:15 +03:00
Evgenii Stratonikov	2ac42b70ce	[#1826] services/object: Parallelize object search ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-07 13:29:15 +03:00
Evgenii Stratonikov	0cb8e7f6f1	[#1837] services/object: Fix log messages ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-07 13:29:15 +03:00
Leonard Lyubich	e54b52ec03	[#1420] object/acl: Fix correlation of object session to request ... In previous implementation of `neofs-node` app object session was not checked for substitution of the object related to it. Also, for access checks, the session object was substituted instead of the one from the request. This, on the one hand, made it possible to inherit the session from the parent object for authorization for certain actions. On the other hand, it covered the mentioned object substitution, which is a critical vulnerability. Next changes are applied to processing of all Object service requests: - check if object session relates to the requested object - use requested object in access checks. Disclosed problem of object context inheritance will be solved within Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-07 10:34:38 +03:00
Evgenii Stratonikov	5834f9807e	[#1847] services/policer: Provide container ID in logs ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-07 09:58:16 +03:00
Evgenii Stratonikov	c20c2e3b39	[#1845] morph/client: Use 0 as OfflineState in update peer ... As the documentation prescribes. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-06 12:26:25 +03:00
Leonard Lyubich	713aea06fa	[#1681] node: Block only Object service ops under maintenance ... In previous implementation node blocked any operation of local object storage in maintenance mode. There is a need to perform some storage operations like data evacuation or restoration. Do not call block storage engine in maintenance mode. Make all Object service operations to return `apistatus.NodeUnderMaintenance` error from each local op. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	eb1fba5182	[#1680] morph/netmap: Adopt to recent contract changes ... After recent Netmap contract changes all read methods which return network map (either candidates or snapshots) encode node descriptors into same structure. Decode `netmap.Node` contract-side structure from the call results. Replace node state with the value from the `netmap.Node.State` field. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	6a2ec21641	[#1681] ir/netmap: Check MAINTENANCE state in UpdateState ... Storage node can be requested to be switched into `MAINTENANCE` state. Inner Ring should accept such requests only if network configuration allows it. Make `Processor` of Netmap contract's notifications to depend on `state.NetworkSettings`. Make `Processor.processUpdatePeer` to call `MaintenanceModeAllowed` if notification event relates to `MAINTENANCE` mode`. Share singe `state.NetworkSettings` provider in Inner Ring application. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	bdb8243a5a	[#1681] morph/netmap: Support MAINTENANCE state notification ... After recent changes Netmap contract can send `UpdateState` notification event with `MAINTENANCE` node's state. There is a need to provide functionality to work with the status. Provide `UpdatePeer.Maintenance` method. Support new state in `ParseUpdatePeer` and `ParseUpdatePeerNotary` functions. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	0903b2af93	[#1680] morph/netmap: Allow to set MAINTENANCE state ... After recent changes in NeoFS protocol storage nodes can be in `MAINTENANCE` state. There is a need to support this state in `UpdateState` operation. Add `UpdatePeerPrm.SetMaintenance` method which makes node to be switched into `MAINTENANCE` mode after the `UpdatePeerState` operation. New functionality is going to be used in Storage node application for Control API serving. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	6b99f2a0df	[#1680] ir/netmap: Correlate node's state with network settings ... Inner Ring should allow registering of storage nodes with `MAINTENANCE` state in the NeoFS network only if its configuration allows this status. Make `networkSettings.MaintenanceModeAllowed` to call `MaintenanceModeAllowed` method of underlying Netmap contract's client in order to assert state allowance. From now nodes will be accepted to the network with `MAINTENANCE` state only with the appropriate network configuration. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	0826310b2a	[#1681] morph/netmap: Support MaintenanceModeAllowed config ... After recent changes in the NeoFS API protocol network configuration contains `MaintenanceModeAllowed` boolean flag. There is a need to support the config value in all NeoFS applications. Provide `Client.MaintenanceModeAllowed` method which read the config from the Sidechain. Extend `NetworkConfiguration` structure with `MaintenanceModeAllowed` field. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	d2d4191868	[#1680] morph/netmap: Support HomomorphicHashingDisabled config ... `NetworkConfiguration` represents NeoFS network configuration stored in the Sidechain. In previous implementation the configuration missed flag of disabled homomorphic hashing. Add `NetworkConfiguration.HomomorphicHashingDisabled` boolean field. Decode the field in `Client.ReadNetworkConfiguration` method. Print this value in `netmap netinfo` command of NeoFS CLI. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	e0194dbde5	[#1681] morph/netmap: Refactor reading the boolean configurations ... `readBoolConfig` method is going to be reused for reading other configuration values. All boolean settings are `false` by default, so it makes sense to return default value on missing key directly from `readBoolConfig`. Handle `ErrConfigNotFound` case in `readBoolConfig` method. Change `HomomorphicHashDisabled` method to call `readBoolConfig` directly. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	f64ae55806	[#1681] ir/netmap: Require MAINTENANCE mode to be allowed by network ... There is a need to prevent limitless abuse of MAINTENANCE status of the storage nodes. To do this, configuration of the NeoFS network is going to be extended with the flag which allows the state. Until this is done, it makes sense to prepare a site for this in the code. Define `state.NetworkSettings` interface as an abstraction of global network configuration within the `state` package. Make `NetMapCandidateValidator` to depend on `NetworkSettings` and provide corresponding field setter. Change `VerifyAndUpdate` method's behavior to return an error for candidates with MAINTENANCE state if this state is disallowed by the network configuration. Provide `NetworkSettings` from the wrapper over Netmap contract's client on Inner Ring application side. The provider is implemented to statically disallow MAINTENANCE mode in order to save previous behavior. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru> Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	42fb40e841	[#1680] morph/netmap: Pre-refactor processing of node states ... New network status of storage nodes is going to be introduced. To simplify the addition, it would be useful to prepare the code for this. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Leonard Lyubich	8858840751	[#1796] ir/netmap: Allow to call AddPeer with ONLINE state only ... In previous implementation Inner Ring allowed storage nodes with any state to register in the network. According to the current design, only nodes with ONLINE state are allowed to enter the network map. Create new `state` sub-package of `nodevalidation` package of Inner Ring application. Define `state.NetMapCandidateValidator` type and provide `NodeValidator` interface required by the Inner Ring's processor of `Netmap` contract's notification events. Embed new validator into the one used by the Inner Ring application. From now all `AddPeer` notifications with node state other than `ONLINE` will be denied. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-10-05 11:41:49 +03:00
Pavel Karpy	4eb0ed11f8	[#1809] node: Do not boot up if metabase is outdated ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 12:32:10 +03:00
Evgenii Stratonikov	8b3b16fe62	[#1825] writecache: Flush cache when moving to the DEGRADED mode ... Degraded mode allows us to operate without an SSD, thus writecache should be unavailable in this mode. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 12:13:09 +03:00
Evgenii Stratonikov	236414df49	[#1817] network: Allow to use network addresses from the iterator ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 12:12:18 +03:00
Evgenii Stratonikov	2e3ef817f4	[#1819] engine: Increase error counter for PUT errors ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 10:11:52 +03:00
Evgenii Stratonikov	b89e71fa78	[#1819] common: Add ErrNoSpace ... Add a common error for this case because it is not an error which should increase error counter. Single error simplifies checks on the call-site. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 10:11:52 +03:00
Evgenii Stratonikov	af56574849	[#1819] engine: Fix error counter in Inhume ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 10:11:52 +03:00
Pavel Karpy	8ebe95747e	[#1770] node: Do not lock on shard's Close call ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Pavel Karpy	887afeaddb	[#1770] engine: Do not lock on shard init ... Init can take a lot of time. Because the mutex is taken, all new operations are blocked. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Pavel Karpy	fbd5bc8c38	[#1770] engine: Support configuration reload ... Currently, it only supports changing the compound of the shards. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Pavel Karpy	5c69e19016	[#1770] fstree: Depth parameter int -> uint64 ... Negative values have no sense. On the other hand it differs from the blobovnicza's configuration and prevents unification. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Pavel Karpy	2d7166f8d0	[#1770] shard: Move NewEpoch event routing on SE level ... It will allow dynamic shard management. Closing a shard does not allow removing event handlers. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Evgenii Stratonikov	1edc048870	[#1697] services/object: Return proper error if session token is missing ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-30 16:13:01 +03:00
Evgenii Stratonikov	0a411908ee	[#1806] writecache: Allow to ignore read errors during flush ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	f2045c10d7	[#1806] shard: Check each component mode when setting mode ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	51b8f26a31	[#1806] services/control: Allow to flush write-cache ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	3d882e9f47	[#1806] engine: Allow to flush write-cache ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	0b4c867ef1	[#1806] writecache: Allow to start flush manually ... Allow user to initiate flushing objects from a writecache. We need this in 2 cases: 1. During writecache storage schema update, it should be flushed with the old version of node and started clean with a new one. 2. During SSD replacement, to avoid data loss. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	55148404ee	[#1806] writecache: Use interfaces for blobstor and metabase ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Leonard Lyubich	93742d37b7	[#1793] node/netmap: Change interface of the latest network map reader ... Replace `ProcessCurrentNetMap` method of `NodeState` interface with `ReadCurrentNetMap` one with two changes: * Replace network map type from NeoFS SDK package with the protocol-generated message. This replaces all the business logic to the application layer. * Support error return. This allows to cover problem node states. Return an error from `NodeState.ReadCurrentNetMap` method implemeted through `atomic.Value` if `Store` method has not been called yet. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-09-27 09:03:17 +03:00
Leonard Lyubich	485a5418d2	[#1793] node: Serve NetmapService.NetmapSnapshot RPC ... There is no more need to serve the same request on Control API. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-09-27 09:03:17 +03:00
Evgenii Stratonikov	6f45cc81fc	[#1764] neofs-node: Use constants for storage types ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-27 08:57:20 +03:00
Evgenii Stratonikov	898689ec14	[#1731] services/replicator: Unify Task interface with other parameters ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-24 13:47:48 +03:00
Evgenii Stratonikov	4e043a801c	[#1731] services/control: Replicate object over network in EvacuateShard RPC ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-24 13:47:48 +03:00
Evgenii Stratonikov	a49137349b	[#1731] engine: Allow to use user handler for evacuated objects ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-24 13:47:48 +03:00
Evgenii Stratonikov	7cc68cf4d4	[#1810] blobovniczatree: Add a test for blobovnicva usage ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-23 10:19:26 +04:00
Evgenii Stratonikov	3df98ce7ba	[#1731] engine: Return the amount of actually moved objects in Evacuate ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-19 11:33:52 +03:00
Evgenii Stratonikov	091d7d30f6	[#1731] services/control: Allow to evacuate data from shard ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-19 11:33:52 +03:00
Evgenii Stratonikov	a51b76056e	[#1731] engine: Add Evacuate command ... Make it possible to move all data from 1 shard to other shards. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-19 11:33:52 +03:00
Evgenii Stratonikov	7377979e12	[#1731] engine: Move single shard PUT to a separate function ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-19 11:33:52 +03:00
Evgenii Stratonikov	5321f8ef9c	[#1786] engine: Unify parameter setters ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-15 10:28:48 +03:00
Evgenii Stratonikov	b064fb24d8	[#1616] engine: Do not use batches in delete ... Use a simple loop at the callsite. This way we remove as much as we can. Also, `Delete` metrics is more meaningful now. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-15 10:28:48 +03:00
Evgenii Stratonikov	cda8f9df2e	[#1786] services/control: Remove WithDeletedObjectHandler option ... Use storage engine directly instead. It is already provided in the options. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-15 10:28:48 +03:00
Evgenii Stratonikov	ae1dab29bc	[#1482] metabase: Encode database keys in binary ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-15 10:28:26 +03:00
Leonard Lyubich	ae655b74f0	*: Exec make fmts ... Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-09-14 17:52:49 +04:00
Pavel Karpy	532c41ca05	[#1658] meta: Do not check object expiration in counters ... Do not perform operations that produce unused results in `syncCounter`. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	431e331373	[#1658] shard: Update metric counters ... Use meta's operation results to change the metrics. Support typed object counters. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	ad47e2a985	[#1658] meta: Add logic counter ... - Meta now supports (and requires) inc/dec labeled counters - The new logic counter is incremented on `Put` operations and is decremented on `Inhume` and `Delete` operations that are performed on _stored_ objects only - Allow force counters sync. "Force" mode should be used on metabase resync and should not be used on a regular meta start Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	d872862710	[#1658] meta: Force counters resync process ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00