1236 commits

Author	SHA1	Message	Date
Dmitrii Stepanov	eec97d177e	[#585] writecache: Count items periodically ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-08-16 14:47:44 +03:00
Evgenii Stratonikov	d15199c5d8	[#596] engine: Consider context errors as logical ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-16 10:39:41 +03:00
Alejandro Lopez	88b6755c5e	[#598] Fix use-after-close bug in badger writecache ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-15 07:04:06 +00:00
Alejandro Lopez	ae8be495c8	[#xx] Avoid manual management of files in tests ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-14 14:01:39 +03:00
Alejandro Lopez	376f03a445	[#598] Hold mode mutex when setting mode ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-14 07:12:27 +00:00
Alexander Chuprov	21800e9fcc	[#162] core: Move literals to constants ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2023-08-11 15:48:42 +03:00
Evgenii Stratonikov	b44a8dd46c	[#597] *: Fix linter warnings ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-11 09:55:51 +03:00
Alejandro Lopez	8f994163ee	[#586] Fix writecache benchmarks and refactor hacky NeedsCompression ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-10 08:05:18 +00:00
Alejandro Lopez	023b90342c	[#584] Disable compression in badger writecache ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-10 10:14:06 +03:00
Evgenii Stratonikov	d641cba2fc	[#587] Do not use math/rand.Read ... Fix staticcheck warnings after go1.20 update. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-09 16:02:44 +03:00
Alejandro Lopez	5b7e4a51b7	[#481] Update frostfs-sdk-go and error pointer receivers ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-09 10:26:53 +00:00
Alejandro Lopez	ae322e9f73	[#576] Set SyncWrites for badger writecache by default ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-09 11:28:26 +03:00
Alejandro Lopez	7da4306e38	[#575] writecache: Fix log level for badger writecache ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-08 08:16:36 +00:00
Dmitrii Stepanov	0e697266c3	[#563] writecache: Fix metrics and bolt delete ... Estimate cache size after delete objects to update metric. Update counters on small object deletion. Do not count bbolt DB file as FSTree object. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-08-07 12:53:28 +00:00
Alejandro Lopez	1a0cb0f34a	[#421] Try using badger for the write-cache ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-07 08:16:57 +00:00
Alejandro Lopez	65c72f3e0b	[#559] Remove manual path handling in fstree tests ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-08-03 10:03:41 +03:00
Evgenii Stratonikov	35dc64bd7b	[#547] metabase: Fix datarace in tests ... Quite an old one bf9e938a3b. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-07-31 15:35:15 +00:00
Evgenii Stratonikov	ec8b4fdc48	[#541] writecache/test: Close writecache on exit ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-07-29 10:57:25 +00:00
Evgenii Stratonikov	ad5f527bd3	[#541] writecache/test: Remove initWC() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-07-29 10:57:25 +00:00
Dmitrii Stepanov	af82c2865e	[#335] treesvc: Fix inmemory unit tests and nil meta items ... Bolt forest saves empty slice of items. Now inmemory forest does it the same way. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-07-20 10:14:10 +03:00
Dmitrii Stepanov	b4e72a2dfd	[#335] treesvc: Sort nodes by Filename in GetSubTree ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-07-20 10:14:10 +03:00
Dmitrii Stepanov	3e8de14e7d	[#382] evacuate: Fix unit tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-07-17 10:24:05 +00:00
Alejandro Lopez	11027945d8	[#479] writecache: Fix writecache fstree flush premature ctx cancel ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-07-14 10:25:52 +03:00
Alexander Chuprov	033eaf77e1	[#496] node: Fix linter importas ... Standardize the alias of the import frostfs-sdk-go/object as objectSDK. Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2023-07-06 15:36:41 +03:00
Evgenii Stratonikov	cab51c8cbe	[#1] metabase: Rename blindlyProcess() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-28 12:13:20 +00:00
Evgenii Stratonikov	f4c71cea65	[#1] *: Replace outdated FIXME/TODO links ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-28 12:13:20 +00:00
Dmitrii Stepanov	43d263c3d5	[#428] linter: Fix unkeyed assignment ... Thanks to gopls. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-26 13:29:39 +00:00
Dmitrii Stepanov	cac4ed93d6	[#428] engine: Add low_mem config parameter ... Concurrent initialization in case of the metabase resync leads to high memory consumption and potential OOM. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-26 13:29:39 +00:00
Evgenii Stratonikov	4bf345225c	[#447] pilorama: Use named constant for the key size ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-26 07:42:05 +00:00
Evgenii Stratonikov	b4ce0b0412	[#447] pilorama: Do not undo log for create ops ... ``` goos: linux goarch: amd64 cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ new │ │ sec/op │ sec/op vs base │ Create-8 36.48µ ± 11% 30.34µ ± 14% -16.84% (p=0.000 n=10) │ old │ new │ │ B/op │ B/op vs base │ Create-8 43.01Ki ± 4% 37.78Ki ± 5% -12.15% (p=0.000 n=10) │ old │ new │ │ allocs/op │ allocs/op vs base │ Create-8 166.0 ± 3% 146.0 ± 3% -12.05% (p=0.000 n=10) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-26 07:42:05 +00:00
Evgenii Stratonikov	dd3874eff1	[#447] pilorama: Add benchmark for create ops ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-26 07:42:05 +00:00
Dmitrii Stepanov	4d48377cec	[#459] blobovniczatree: Fix get error ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 16:44:53 +03:00
Dmitrii Stepanov	b5d9f4a285	[#373] metrics: Add pilorama metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	e89fa110c7	[#373] metrics: Add metabase metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	af608da952	[#373] metrics: Add blobovnizca metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	56f320dd85	[#373] metrics: Add blobstor metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	16a142cd0c	[#373] metrics: Add FSTree metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	d8ecc69d00	[#373] local storage: Pass parent ID ... This is required to add shard ID as metric label. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	d5aaec1107	[#373] pilorama: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	059e9e88a2	[#373] metabase: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	f54cc0b607	[#373] blobstor: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	8318d90ad0	[#373] blobovniczatree: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	3ae3c8dfdb	[#373] fstree: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	a8526d45e9	[#373] blobovnizca: Add missed/fix tracing spans ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	028d4a8058	[#373] blobovnicza: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-21 15:13:26 +03:00
Dmitrii Stepanov	01a0c97760	[#453] engine: Set Disabled mode to deleted shard ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-20 12:04:07 +03:00
Evgenii Stratonikov	69df0d21c2	[#446] engine: Move to read-only on blobstor errors ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-16 14:53:32 +03:00
Evgenii Stratonikov	fe01781811	[#446] los: Wrap SSD errors in a separate type ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-16 14:53:31 +03:00
Evgenii Stratonikov	20b84f183a	[#446] engine: Simplify logs for shard mode change ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-16 14:51:29 +03:00
Dmitrii Stepanov	4449006862	[#424] metrics: Use mode value as metric value for shard ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 18:26:19 +03:00
Dmitrii Stepanov	1b364d8cf4	[#424] metrics: Refactor engine metrics ... Use histogram vector to measure request duration. Fix naming like in Prometheus best practice. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:53:32 +03:00
Dmitrii Stepanov	85deb12f4d	[#424] writecache: Drop metrics when close ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:52:40 +03:00
Dmitrii Stepanov	07f155ac77	[#424] metrics: Use labels for writecache methods and operations ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-14 14:52:40 +03:00
Dmitrii Stepanov	2541d319de	[#266] pilorama: Allow to get current tree height ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-13 10:00:45 +00:00
Evgenii Stratonikov	41ab4d070e	[#423] *: Use hrw.StringHash() where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-06-13 07:18:25 +00:00
Anton Nikiforov	263c6fdc50	[#372] node: Add metrics for the error counter in the engine ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-06-07 13:04:47 +00:00
Dmitrii Stepanov	74578052f9	[#412] node: Replace tracing package ... Use observability module. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-06-01 13:23:11 +00:00
Dmitrii Stepanov	dbf41391b5	[#401] engine: Extend evacuation logs ... Add operation-tag to logger. Log evacuation results. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-31 13:24:30 +03:00
Dmitrii Stepanov	3220c4df9f	[#376] metrics: Add GC metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-31 10:22:12 +00:00
Alejandro Lopez	faca861451	[#411] Remove unnecessary pointers for sync objects ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-31 10:19:14 +00:00
Anton Nikiforov	8dcd06c587	[#394] node: Use Context in Blobovniczas.Iterate() ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-31 10:09:18 +00:00
Anton Nikiforov	365a7ca0f4	[#366] node: Stop GC once termination signal received ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-29 09:35:08 +03:00
Anton Nikiforov	802168c0c6	[#364] node: Stop flushing big object when termination signal received ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-05-26 16:46:58 +03:00
Dmitrii Stepanov	271a56c2ab	[#395] metrics: Drop redundant metrics ... HistogramVec already has labeled counter. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-26 13:39:37 +00:00
Evgenii Stratonikov	f2e5dead7e	[#398] pilorama: Disallow applying same operations ... 1. In redo() we save the old state. 2. If we do redo() for the same operation twice, the old state will be overritten with the new one. 3. This in turn affects undo() and subsequent isAncestor() check. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-26 12:14:29 +00:00
Dmitrii Stepanov	20a489bdb5	[#393] gc: Use defer to mark handler done ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-26 12:14:02 +00:00
Dmitrii Stepanov	2613351008	[#387] gc: Cancel GC is change mode requested ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-25 09:38:16 +03:00
Dmitrii Stepanov	2ce43935f9	[#312] metrics: Add writecache metrcis ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 10:18:39 +00:00
Dmitrii Stepanov	d212d908b5	[#312] wc: Add metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 10:18:39 +00:00
Dmitrii Stepanov	4503a61997	[#312] wc: Delete unused Iterate method ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-24 10:18:39 +00:00
Evgenii Stratonikov	4b768fd115	[#381] *: Move to sync/atomic ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-23 08:18:01 +03:00
Evgenii Stratonikov	ff570847a4	[#381] go.mod: Update bbolt ... Adopt new `ForEachBucket` function where possible. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-22 11:49:14 +03:00
Evgenii Stratonikov	35c9b6b26d	[#314] writecache: remove objects right after they are flushed ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-19 09:37:05 +00:00
Pavel Karpy	bf79d06f03	[#314] writecache: Do not lose small objects on disk errors ... Do return error if an object could not been stored on WC's disk. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-05-19 09:37:05 +00:00
Pavel Karpy	9e56592be3	[#314] writecache: Simplify background workers naming ... Also, drop not used arg. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-05-19 09:37:05 +00:00
Dmitrii Stepanov	e4889e06ba	[#329] node: Make evacuate async ... Now it's possible to run evacuate shard in async. Also only one evacuate process can be in progress. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-19 08:43:52 +00:00
Dmitrii Stepanov	869fcbf591	[#332] gc: Fix expired complex object deletion ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-16 12:44:57 +00:00
Dmitrii Stepanov	ab07bad33d	[#332] gc: Add complex object unit test ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-16 12:44:57 +00:00
Evgenii Stratonikov	c62025c836	[#321] metabase/test: execute tests in parallel ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	945454f60c	[#321] engine/test: Execute tests in parallel ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	4578d00619	[#321] shard/test: Execute tests in parallel ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	d35e4c389f	[#321] shard/test: Parallelize TestWriteCacheObjectLoss ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Evgenii Stratonikov	969bfb603f	[#321] shard/test: Parallelize TestShard_List ... ``` go test -count=1 -run TestShard_List -race . Before: 2.492s After: 0.109s ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-12 09:45:03 +00:00
Dmitrii Stepanov	a181c9e434	[#332] gc: Add additional logging ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-10 17:34:40 +03:00
Alejandro Lopez	973af12854	[#327] tests: replace os.MkdirTemp with t.TempDir ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-05-05 11:24:51 +00:00
Evgenii Stratonikov	a1b4ba9980	[#209] compression: Do not store uncompressible data ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-05-03 16:28:29 +03:00
Dmitrii Stepanov	eca5c210dd	[#299] evacuate: Add context cancel checks ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-05-02 15:55:30 +03:00
Evgenii Stratonikov	a9c4ba62c3	[#248] metabase: Remove storage group bucket ... Backwards compatible change, so no version increase. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 17:00:43 +03:00
Evgenii Stratonikov	58f1ba4b51	[#288] pilorama: Add missing operation in log ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Evgenii Stratonikov	daa26f6e9b	[#288] pilorama/test: Check operation order for TreeGetByPath() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Evgenii Stratonikov	291f9e809a	[#288] pilorama: Remove getMeta() wrapper ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Evgenii Stratonikov	0045f1bcd4	[#288] pilorama: Use more descriptive names for memory tree ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Evgenii Stratonikov	f856ad7480	[#288] pilorama: Remove childMap from memory forest ... Memory forest is here to check the correctness of boltdb optimized implementation. Let's keep it simple. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-28 07:56:20 +00:00
Airat Arifullin	ada081dfd5	[#19] node: Make policier read shards concurrently ... * Introduce ListWithMultiCursor that simultaneously reads objects from different shards Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-27 11:11:47 +03:00
Evgenii Stratonikov	1f4061c0e2	[#285] blobonicza: Optimize upperPowerOfTwo() ... The real reason is this: ``` pkg/local_object_storage/blobovnicza/sizes.go:36:69 revive empty-block: this block is empty, you can remove it ``` Didn't want to make this function longer or to add `nolint`, thus this change. To justify: ``` UpperBound/size=1-8 0.4924n ± 1% 0.2472n ± 2% -49.80% (p=0.000 n=10) UpperBound/size=1023-8 0.4936n ± 3% 0.2442n ± 1% -50.52% (p=0.000 n=10) UpperBound/size=66560-8 0.8201n ± 2% 0.2436n ± 1% -70.29% (p=0.000 n=10) UpperBound/size=41943040-8 6.6900n ± 5% 0.2432n ± 0% -96.36% (p=0.000 n=10) geomean 1.075n 0.2446n -77.24% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-26 17:35:11 +03:00
Evgenii Stratonikov	f07e2d4812	[#285] lint: Fix revive/unused-parameter ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-26 17:35:04 +03:00
Pavel Karpy	09938a9841	Revert "[#262] meta: Do not return old expired objects" ... This reverts commit 3d23b087 Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-20 19:22:55 +03:00
Pavel Karpy	e9461686b8	[#274] wc: Resolve possible deadlock ... If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is called immediately) there is a race and a deadlock that do not allow finish (and start, in fact) an initialization routine because of taken `modeMtx` and also do not allow finish `Close` call because of awaiting initialization finish. So do stop initialization _before_ any mutex is taken. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-20 19:22:54 +03:00
Pavel Karpy	6b6f33ed71	[#274] wc: Make wait groups work more explicit ... Do not run routine that calls `wg.Done()` inside, it is hard to read. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-20 19:22:54 +03:00
Airat Arifullin	6c90bb87f1	[#118] node: add ctx for unit tests for blobstor ... Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-19 18:09:33 +03:00
Pavel Karpy	3d23b08773	[#262] meta: Do not return old expired objects ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-19 13:37:38 +00:00
Airat Arifullin	13c8afcb02	[#118] node: add unit concurrent tests for blobstor ... Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-19 10:22:50 +00:00
Pavel Karpy	20cd080323	[#255] write-cache: Fix init race condition ... Do not use WC's internals in the initialization routines without mode protection. WC should be able to change its mode even if the initialization is not finished yet. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-19 09:29:07 +00:00
Pavel Karpy	262c9c2b93	[#256] blobovniczaTree: Make Exists test stable ... Corrupt and request _the same_ file. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-04-14 16:46:24 +03:00
Evgenii Stratonikov	8466894fdf	[#250] control: remove DumpShard and RestoreShard RPC ... We have `Evacuate` with a cleaner interface. Also, remove them from CLI and engine. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-14 12:28:49 +00:00
Dmitrii Stepanov	6121b541b5	[#242] treesvc: Add tracing spans ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-14 10:25:53 +00:00
Dmitrii Stepanov	d62c6e4ce6	[#242] node: Add tracing spans ... Add tracing spans for PUT requests. Add tracing spans for DELETE requests. Add tracing spans for SELECT requests. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-14 10:25:53 +00:00
Alejandro Lopez	41eb3129ae	[#139] Refactor blobovnicza exist test to not use chmod ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-04-14 12:16:14 +03:00
Evgenii Stratonikov	0e31c12e63	[#240] logs: Move log messages to constants ... Drop duplicate entities. Format entities. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-14 05:06:09 +00:00
Evgenii Stratonikov	5d2affa5cd	testutil: Fix linter warning ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-12 11:00:02 +03:00
Dmitrii Stepanov	0920d848d0	[#135] get-object: Add tracing spans ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-12 06:52:00 +00:00
Evgenii Stratonikov	c4865783fc	[#236] blobstor/test: Prefill storage in parallel in read benchmark ... `blobovniczatree` takes a really long time to prefill, because each batch takes at least 10ms, so for 10k iterations we have at least 100s of prefill. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-11 20:52:11 +03:00
Evgenii Stratonikov	6ad5c38225	[#236] testutil: Use random object id in RandObjGenerator ... Before this commit it was like this: ``` BenchmarkSubstorageWritePerf/memstore-rand10-8 227425 4859 ns/op BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8 perf_test.go:165: writing entry: file exists perf_test.go:165: writing entry: file exists perf_test.go:165: writing entry: file exists BenchmarkSubstorageWritePerf/fstree-rand10-8 --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8 perf_test.go:165: writing entry: file exists perf_test.go:165: writing entry: file exists perf_test.go:165: writing entry: file exists ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-11 20:52:04 +03:00
Evgenii Stratonikov	c85a0bc866	[#236] blobstor/test: Reduce test descriptions ... I tried to add 4 more tests and suddenly, it became harder to navigate in code. Move directory creation in a common function. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-11 20:52:04 +03:00
Evgenii Stratonikov	dbc3811ff4	[#191] engine: Allow to remove redundant object copies ... RemoveDuplicates() removes all duplicate object copies stored on multiple shards. All shards are processed and the command tries to leave a copy on the best shard according to HRW. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-07 17:25:50 +00:00
Airat Arifullin	cb172e73a6	[#228] node: Use uber atomic package instead standard ... Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-07 15:37:27 +00:00
Dmitrii Stepanov	f32f61df87	[#203] pilorama: Refactor tests ... Do not pass 0 as channel capacity. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Dmitrii Stepanov	8908798f59	[#203] node: Resolve unused vars ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Dmitrii Stepanov	bab11492ad	[#203] node: Resolve never used errors ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Dmitrii Stepanov	9e2df4b7c7	[#203] node: Fix double imports ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-06 16:33:36 +03:00
Airat Arifullin	ab891517de	[#116] node: Fix bug with extra generated files in TestReload ... * Create testNewEngine in engineWithShards without default opts Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-05 18:49:34 +03:00
Airat Arifullin	6f7b6a8813	[#116] node: Improve shard/engine construction in tests ... * Introduce testEngineWrapper that can be constructed with different options Signed-off-by: Airat Arifullin a.arifullin@yadro.com	2023-04-05 14:36:40 +00:00
Dmitrii Stepanov	23575e1ac0	[#210] policier: Resolve contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-05 14:55:52 +03:00
Evgenii Stratonikov	9098d0eec0	[#211] engine: Unify shard mode checks for tree operations ... All operations must ensure the shard is not in a degraded mode. Write operations must also ensure the shard is not in a read-only mode. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-05 11:10:39 +00:00
Evgenii Stratonikov	760af6b912	[#211] fstree: Consider ENOSPC a logical error ... We already do this for file writing, however directory creation can also fail. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-04-05 11:10:39 +00:00
Dmitrii Stepanov	1f1aed87be	[#188] metabase: Refactor object inhume ... Resolve funlen linter for db.Inhume method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	5a66db80c5	[#188] engine: Refactor shard evacuation ... Resolve funlen and gocognit linter for StorageEngine.Evacuate method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	456bc097f7	[#188] engine: Refactor get range from engine ... Resolve funlen linter for StorageEngine.getRange method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	3010ca2649	[#188] engine: Refactor get object from engine ... Resolve funlen linter for StorageEngine.get method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	0739c36a3b	[#188] metabase: Refactor object put to metabase ... Resolve funlen linter for db.put method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	8273a3dfb2	[#188] blobstor: Refactor blobstor test ... Resolve funlen linter for TestIterate function Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	594b5821ed	[#188] blobstor: Refactor put data to blobovniczas ... Resolve funlen linter for Blobovniczas.Put method Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 14:50:43 +03:00
Dmitrii Stepanov	8e5a0dcf27	[#204] gc: Fix GC handlers start ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-04-04 06:48:27 +00:00
Dmitrii Stepanov	a7c79c773a	[#168] node: Refactor node config ... Resolve containedctx linter for cfg Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-31 09:32:59 +03:00
Airat Arifullin	7be5a0fd79	[#189] node: Fix bug with sync/atomic package ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2023-03-30 14:50:43 +00:00
Dmitrii Stepanov	ab32067152	[#183] gc: Fix drop expired locked complex objects ... Do not delete bucket keys during iteration Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 15:33:42 +03:00
Dmitrii Stepanov	9f0bce5c15	[#183] gc: Fix drop expired locked simple objects ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-30 15:33:42 +03:00
Alejandro Lopez	341fe1688f	[#139] test: Add test storage implementation ... This aims to reduce the usage of chmod hackery to induce or simulate OS-related failures. Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-03-29 14:28:49 +00:00
Pavel Karpy	97e201993b	[#184] *: Unify done contexts handling ... If `ctx.Done()`, return `ctx.Err()` in every function that returns an error. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-29 10:34:54 +00:00
Airat Arifullin	221203beeb	[#180] node: Refactor panics in unit test ... * Replace panics in unit tests by require.NoError and t.Fatalf Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2023-03-29 12:39:07 +03:00
aarifullin	34329d67ff	[#86] node: Fix unit test and linter errors ... Signed-off-by: Airat Arifullin <aarifullin@yadro.com>	2023-03-23 12:42:58 +03:00
Ayrat Arifullin	9808dec591	[#86] node: Move testing utils to one package ... Move testing utils from tests in local_object_storage package to unified testutil package Signed-off-by: Airat Arifullin <aarifullin@yadro.com>	2023-03-23 08:19:15 +00:00
Denis Kirillov	342e571d89	[#159] Add handle __SYSTEM__ sys attributes ... Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>	2023-03-22 17:35:20 +03:00
Pavel Karpy	da8da1c63a	[#98] fstree: Do not fail iteration over just removed files ... A directory is read and files are saved to a local variable. The iteration over such files may lead to a non-existing files reading due to a normal SN operation cycle and, therefore, may lead to a returning the OS error to a caller. Skip just removed (or lost) files as the golang std library does in similar situations: 5f1a0320b9/src/os/dir_unix.go (L128-L133). Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-22 07:26:27 +00:00
Evgenii Stratonikov	8014fdb21a	[#156] metabase: Make freePotentialLocks() pass linter checks ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Evgenii Stratonikov	fb13902db9	[#156] shard: Make refillMetabase() pass linter checks ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Evgenii Stratonikov	47e8c5bf23	[#156] pilorama: Remove CIDDescriptor from TreeApply() ... Initially it was there to check whether an update is being initiated by a proper node. It is now obsolete for 2 reasons: 1. Background synchronization fetches all operations from a single node. 2. There are a lot more problems with trust in the tree service, it is only used in controlled environments. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-22 07:14:18 +00:00
Dmitrii Stepanov	5059dcc19d	[#145] shard-gc: Delete expired objects after locks ... GC deletes expired locks and objects sequentially. Expired locks and objects are now being deleted concurrently in batches. Added a config parameter that controls the number of concurrent workers and batch size. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 11:31:08 +03:00
Dmitrii Stepanov	6c4a1699ef	[#145] shard-gc: Expired locked unit test ... Added unit test that verifies that GC deletes expired locked objects in one epoch. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 11:31:08 +03:00
Dmitrii Stepanov	481a1ca6f3	[#148] linter: Add gocognit linter ... Code with high cognitive complexity is hard intuitively to understand Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 09:54:41 +03:00
Dmitrii Stepanov	97c36ed3ec	[#148] linter: Add funlen linter ... Long functions are hard to understand and source of errors Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 09:54:41 +03:00
Dmitrii Stepanov	2dc86058c3	[#148] memstore: Drop space line ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-03-21 09:52:39 +03:00
Pavel Karpy	f006f3b342	[#67] node: Make engine's IsLocked public ... It will allow reusing that method in expiration checks. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-16 16:20:45 +03:00
Alejandro Lopez	724debfdcd	[#81] node: Add basic read/write benchmarks for substorages ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-03-15 16:37:04 +00:00
Evgenii Stratonikov	3e6fd4c611	[#82] pilorama: Allow to store last sync height ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-13 11:25:44 +00:00
Evgenii Stratonikov	861e9ab59a	[#83] pre-commit: Add initial configuration ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-03-13 07:07:29 +00:00
Pavel Karpy	f1f3c80dbf	[#32] node: Init write-cache asynchronously ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-09 11:07:33 +00:00
Pavel Karpy	381e363a8b	[#32] node: Always close general components after testing ... It will prevent test fails with `-race` flag on components that have background processes and make some actions on test framework. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-03-09 11:07:33 +00:00
Alex Vanin	20de74a505	Rename package name ... Due to source code relocation from GitHub. Signed-off-by: Alex Vanin <a.vanin@yadro.com>	2023-03-07 16:38:26 +03:00
Anton Nikiforov	e9f3c24229	[#65] Use strings.Cut instead of strings.Split* where possible ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-02-28 13:39:14 +03:00
Dmitrii Stepanov	6925fb4c59	[TrueCloudLab/hrw#2] node: Use typed HRW methods ... Update HRW lib and use typed HRW methods to sort shards and nodes Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-02-28 13:36:25 +03:00
Dmitrii Stepanov	c3a7039801	[TrueCloudLab/hrw#2] node: Optimize shard hash ... Compute shard hash only once Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-02-28 13:36:25 +03:00
Alejandro Lopez	cb5468abb8	[#66] node: Replace interface{} with any ... Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>	2023-02-21 16:47:07 +03:00
Pavel Karpy	337049b2ce	[#56] node: Allow reading expired locked object ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-21 09:56:57 +03:00
Pavel Karpy	3beef10f89	[#61] node: Do not fetch missing objects ... If an object is missing in a `meta`, shard should not look for it in a `blobstor`. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 14:47:38 +03:00
Evgenii Stratonikov	d1d123d180	[#2234] writecache: Fix possible panic in initFlushMarks ... In case we have many small objects in the write-cache, `indices` should not be reused between iterations. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	315141dc2c	[#2252] fstree: Allow concurrent writes ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Pavel Karpy	07ec51ea60	[#2244] node: Add object address to WC's operations ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 13:53:27 +03:00
Pavel Karpy	dbbbef9ddb	[#2244] node: Update expired storage ID by WC ... Previously, node could get an "infinite" small object: it could be expired and thus could not be flushed (update its storage ID) to metabase => could not be marked as flushed => node never removes such object and repeat all the cycle one more time. If object exists and is not marked with GC (meta returns `ErrObjectIsExpired`, not `ObjectNotFound` and not `ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt transaction. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	5cb2c5ae62	[#2238] engine: Add test for component initialization failures ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	427fe276f2	[#2238] shard: Try closing all components ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	c53903ccd0	[#2238] engine: Make Open and Init similar ... 1. Both could initialize shards in parallel. 2. Both should close shards after an error. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	e0309e398c	[#2239] writecache: Fix possible deadlock ... LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction. `View` transaction itself takes `mmapLock` [1], which is lifted after tx finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`) When we evict items from LRU cache mutex order is different: first we take LRU mutex and then execute `Batch` which _does_ take `mmapLock` in case we need to remap. Thus the deadlock. [1] 8f4a7e1f92/db.go (L708) Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Evgenii Stratonikov	58367e4df6	[#2232] pilorama: Merge in-queue batches ... To achieve high performance we must choose proper values for both batch size and delay. For user operations we want to set low delay. However it would prevent tree synchronization operations to form big enough batches. For these operations, batching gives the most benefit not only in terms of on-CPU execution cost, but also by speeding up transaction persist (`fsync`). In this commit we try merging batches that are already _triggered_, but not yet _started to execute_. This way we can still query batches for execution after the provided delay while also allowing multiple formed batches to execute faster. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-20 13:53:27 +03:00
Pavel Karpy	40822adb51	[#2213] node: Do not return object expired object ... "Object is expired" means that object is presented in `meta` but it is not `ObjectNotFound` error. Previous implementation made `shard` search for an object without `meta` which was an error. Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-20 13:53:27 +03:00
Artem Tataurov	362f24953a	[#47] shard: Switch container size metric from physical to logical capacity ... Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>	2023-02-17 12:03:42 +03:00
Evgenii Stratonikov	204cd3a11c	[#31] fstree: Optimize treePath ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-10 12:49:31 +03:00
Evgenii Stratonikov	dee4498c1e	[#31] fstree: Do not check for a file existence twice ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-10 12:49:31 +03:00
Evgenii Stratonikov	abbecf49d6	[#31] fstree: Speedup string-to-address conversion ... ``` name old time/op new time/op delta _addressFromString-8 1.25µs ±30% 1.02µs ± 6% -18.49% (p=0.000 n=9+9) name old alloc/op new alloc/op delta _addressFromString-8 352B ± 0% 256B ± 0% -27.27% (p=0.000 n=9+10) name old allocs/op new allocs/op delta _addressFromString-8 6.00 ± 0% 4.00 ± 0% -33.33% (p=0.000 n=10+10) ``` Also, assure compiler that `s` doesn't escape: Before this commit: ``` ./fstree.go:74:24: leaking param: s ./fstree.go:90:6: moved to heap: addr ``` After this commit: ``` ./fstree.go:74:24: s does not escape ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-02-10 12:49:31 +03:00
Artem Tataurov	ab21d90cfb	[#1794] shard: Add increasing case for the payload size metric ... Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>	2023-02-09 13:30:23 +03:00
Stanislav Bogatyrev	cb016d53a6	[#1] Fix comments and error messages ... Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>	2023-02-06 17:41:14 +03:00
Pavel Karpy	73bc1b0b68	[#38] node: Fix linter warnings ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-02-06 17:27:54 +03:00
Pavel Karpy	89a0266f5e	[#1794] metrics: Track physical object capacity per shard ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-01-26 20:06:28 +03:00
Evgenii Stratonikov	9513f163aa	[#2116] metrics: Track physical object capacity in the container ... Currently we track based on `PayloadSize`, because it is already stored in the metabase and it is easier to calculate without slowing down the whole system. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com> Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2023-01-26 20:06:28 +03:00
Evgenii Stratonikov	d65a95a2c6	[#28] pilorama: Remove LogMove struct ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	c72576e72f	[#2208] engine: Log time-consuming shard operations ... Currently the only way to tell whether `evacuate/set-mode` is finished is to set a very big timeout and _hope_ that the operation will finish. In this commit we add INFO logs for such operations which should simplify the life of an administrator. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	87f0e3ea25	[#2208] fstree: Rename file after write ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	792319a044	[#2208] fstree: Remove file if there was an error during write ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	25d5995cef	[#2210] pilorama: Allocate bucket name outside of batches ... 1. Reduce allocations inside transactions. 2. Do not encode container ID to string: it allocates a lot and takes more space. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	165a600624	[#2210] pilorama: Reduce the amount of keys per node ... Under high load we are limited by the _amount_ of keys we need to update in a single transaction. In this commit we try storing all state with a single key. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Pavel Karpy	64a5294b27	[#2200] shard: Do not fetch big objects from blobovniczas ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Pavel Karpy	91757329ae	[#2200] shard: Fix blobstor obj fetching ... In the previous implementation any non-nil error that preceded object fetching from blobstor led to iterating over every storage (in other words, no storage ID information was taken into account). Now storage ID is skipped only if metabase (storage ID source) returns any error. Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Pavel Karpy	cf1a91a758	[#2206] blobovnicza: Use Latin letters in the code ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	6451f019d2	[#2203] shard: Do not panic in Close after unsuccessful Init ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	ac81c70c09	[#1621] pilorama: Batch related operations ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	9009612a82	[#2198] blobovniczatree: Properly handle concurrent active blobovnicza update ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	cedbd380f2	[#2197] pilorama: Close database in degraded mode ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	b0ad1b9ed2	[#2193] pilorama: Use do in TreeMove ... It should be similar to a `TreeAddByPath`. `applyOperation` is used for `Apply` when the operation can be inserted in the middle of a log. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Evgenii Stratonikov	ba393e3e91	[#2188] engine: Fix panic during setting shard mode ... Under load changing shard mode can lead to it being removed from the list during some other PUT. ``` Dec 28 07:01:26 az neofs-node[364505]: panic: runtime error: invalid memory address or nil pointer dereference Dec 28 07:01:26 az neofs-node[364505]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc9fbb1] Dec 28 07:01:26 az neofs-node[364505]: goroutine 11791912 [running]: Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).putToShard(0xc000435490, {0xc0003f7a28?, 0xc0001192c0?}, 0x2, {0x0, 0x> Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:91 +0x1b1 Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put.func1(0xc000435490?, {0xc0003f7a28?, 0xc0001192c0?}) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:71 +0x19c Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).iterateOverSortedShards(0x1?, {{0x62, 0x23, 0xfe, 0x60, 0x67, 0xd5, 0x> Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:225 +0xc8 Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put(0xc000435490, {0x1?}) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:66 +0x2a9 Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put.func1() Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:43 +0x2a Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).execIfNotBlocked(0x8?, 0x38?) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:147 +0xcf Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put(0xc4df775a80?, {0x0?}) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:42 +0x65 Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.Put(0xc06d928b80?, 0xc06b1b8dc8?) Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:158 +0x19 Dec 28 07:01:26 az neofs-node[364505]: main.engineWithoutNotifications.Put({0x20301b?}, 0x20301b?) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-01-25 15:31:47 +03:00
Anton Nikiforov	3d57f4c961	[#2179] test: Fix test TestEvacuateNetwork/multiple_shards,_evacuate_many ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2023-01-24 13:37:49 +03:00
Evgenii Stratonikov	9936b112b8	[#5] blobstor: Use generic LRU cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	4155c1bdff	[#5] writecache: Use generic LRU cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-31 23:04:06 +03:00
Evgenii Stratonikov	0272218eb9	[#2184] compression: Properly calculate upper bound ... If the data is not compressible allocating `len(data)` will lead to a slice reallocation. For a compressible data the results for small size are flaky and we allocate a bit more. However, it feels right to use a provided function if we need to pick any size at all. ``` name old time/op new time/op delta Compression/size=128/zeroed_slice-8 2.23µs ±12% 2.06µs ± 6% -7.35% (p=0.009 n=10+10) Compression/size=128/not_so_random_slice_(block_=_123)-8 19.0µs ±10% 15.8µs ±16% -17.09% (p=0.000 n=9+10) Compression/size=128/random_slice-8 17.6µs ±15% 16.1µs ±16% ~ (p=0.075 n=10+10) Compression/size=1024/zeroed_slice-8 3.05µs ±11% 2.84µs ±10% ~ (p=0.089 n=10+10) Compression/size=1024/not_so_random_slice_(block_=_123)-8 18.1µs ± 6% 18.2µs ±12% ~ (p=0.971 n=10+10) Compression/size=1024/random_slice-8 48.6µs ± 6% 45.6µs ± 5% -6.07% (p=0.006 n=10+9) Compression/size=32768/zeroed_slice-8 26.8µs ± 3% 28.7µs ± 8% +7.23% (p=0.001 n=10+10) Compression/size=32768/not_so_random_slice_(block_=_123)-8 44.3µs ± 8% 43.7µs ±13% ~ (p=0.762 n=8+10) Compression/size=32768/random_slice-8 97.3µs ±32% 68.9µs ±15% -29.13% (p=0.000 n=10+10) Compression/size=33554432/zeroed_slice-8 29.8ms ± 9% 30.3ms ±17% ~ (p=1.000 n=9+9) Compression/size=33554432/not_so_random_slice_(block_=_123)-8 33.1ms ±14% 30.3ms ±11% -8.61% (p=0.043 n=10+10) Compression/size=33554432/random_slice-8 41.7ms ± 3% 30.1ms ± 8% -27.72% (p=0.000 n=9+10) name old alloc/op new alloc/op delta Compression/size=128/zeroed_slice-8 128B ± 0% 144B ± 0% +12.50% (p=0.000 n=10+10) Compression/size=128/not_so_random_slice_(block_=_123)-8 384B ± 0% 144B ± 0% -62.50% (p=0.000 n=10+10) Compression/size=128/random_slice-8 384B ± 0% 144B ± 0% -62.50% (p=0.000 n=10+10) Compression/size=1024/zeroed_slice-8 1.02kB ± 0% 1.15kB ± 0% +12.50% (p=0.000 n=10+10) Compression/size=1024/not_so_random_slice_(block_=_123)-8 1.02kB ± 0% 1.15kB ± 0% +12.50% (p=0.000 n=10+10) Compression/size=1024/random_slice-8 2.56kB ± 0% 1.15kB ± 0% -55.00% (p=0.000 n=10+10) Compression/size=32768/zeroed_slice-8 32.8kB ± 0% 41.0kB ± 0% +25.00% (p=0.000 n=10+10) Compression/size=32768/not_so_random_slice_(block_=_123)-8 32.8kB ± 0% 41.0kB ± 0% +25.00% (p=0.000 n=10+10) Compression/size=32768/random_slice-8 81.9kB ± 0% 41.0kB ± 0% -50.00% (p=0.000 n=10+10) Compression/size=33554432/zeroed_slice-8 33.6MB ± 0% 33.6MB ± 0% +0.02% (p=0.000 n=9+9) Compression/size=33554432/not_so_random_slice_(block_=_123)-8 33.6MB ± 0% 33.6MB ± 0% +0.02% (p=0.000 n=8+10) Compression/size=33554432/random_slice-8 75.5MB ± 0% 33.6MB ± 0% -55.55% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Compression/size=128/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=128/not_so_random_slice_(block_=_123)-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) Compression/size=128/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) Compression/size=1024/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=1024/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=1024/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) Compression/size=32768/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=32768/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=32768/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) Compression/size=33554432/zeroed_slice-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=33554432/not_so_random_slice_(block_=_123)-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) Compression/size=33554432/random_slice-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	0ace28e43d	[#2175] blobovniczatree: Close all non-active blobovniczas ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	c1cf418956	[#2175] blobovniczatree: Make function parameters more descriptive ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	b4e90cdf51	[#2165] pilorama: Optimize TreeApply when used for synchronization ... Because synchronization _most likely_ will have apply already existing operations, it is much faster to check their presence in a read transaction. However, always doing this will degrade the perfomance for normal `Apply`. And, let's be honest, it is already not good. Thus we add a separate parameter which specifies whether this logic is enabled. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	21717262ec	[#2016] shard: Check meta first on Get ... `meta` should prevent returning removed objects (`GCMark` and `TS` relations are `meta` abstractions). Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	74ec71446f	[#2167] shard: Do not use write-cache by default in Head ... Both `meta` and `write-cache` are expected to have a fast underlying disk, so it does not seem like an optimisation. Moreover, `write-cache`'s `Head` is a `Get` with payload cutting, it _must_ use more memory for no reason (`meta` was created for such requests). Also, `write-cache` does not allow performing any "meta" relations checks (such as locking, tombstoning). Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	1608fd1c07	[#2167] write-cache: Add "write-cache" to its logs ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	eea2892109	[#1956] node: Lock shard's mode on its methods switch ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	e1c3bdbfa6	[#1621] pilorama: Remove Timestamp field from nodeInfo ... It is already present in `Meta`. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	1044adbe94	[#1621] pilorama: Improve memory allocation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	2539d466a6	[#1621] pilorama: Seek after cursor invalidation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	e9ba8931f8	[#1621] pilorama: Simplify bucket creation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	fe7ddfdc6a	[#1621] pilorama: Compare memory forests properly ... Node children are not sorted and could occur in any order. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-30 11:07:35 +03:00
Anton Nikiforov	edb1428248	[#2022] Add metric readonly to get shards mode ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	e5c304536b	[#2161] pilorama: Do not apply already existing operations ... Speeds up synchronization a bit. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	21c58c92a9	[#2145] meta: Do allow force inhuming a locked object ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-30 11:07:35 +03:00
Evgenii Stratonikov	0b78af467e	[#2140] engine: Fix error handling in TreeMove ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2022-12-30 11:07:35 +03:00
Pavel Karpy	923f84722a	Move to frostfs-node ... Signed-off-by: Pavel Karpy <p.karpy@yadro.com>	2022-12-28 15:04:29 +03:00
Evgenii Stratonikov	42554a9298	[#2068] writecache: Remove deleted objects from the writecache ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Evgenii Stratonikov	4a49ea0855	[#2068] writecache: Allow to open FSTree in read-only mode ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Evgenii Stratonikov	857d2dc3f5	[#2068] writecache: Optimize initial flush existence checking ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Evgenii Stratonikov	63f604e948	[#2068] blobstor: Allow to provide storage ID in Exists ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Evgenii Stratonikov	6ad2b5d5b8	[#2068] blobovnicza: Add Exists method ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-12-02 11:52:05 +03:00
Pavel Karpy	3d0768a1d3	[#2061] node: Unify meta.Get benchmarks ... Make them get exactly one (different) object per a bench iteration. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 18:29:14 +03:00
Pavel Karpy	bc905f169d	[#2061] meta: Add parallel bench for Get ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-30 18:29:14 +03:00
Evgenii Stratonikov	7335a52f29	[#1732] pilorama: Improve logical error handling ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-30 16:58:52 +03:00
Evgenii Stratonikov	ae7b473768	[#2064] blobovniczatree: Remove index too big log ... There is no need to log about a situation which is expected. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-30 16:53:18 +03:00
Pavel Karpy	ed4351aab0	[#2074] write-cache: Do not flush same object twice ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	dd225906a0	[#2074] write-cache: Remove unused variables ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	b1025bdb42	[#2057] meta: Fail write operations in R/O mode ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	fdeea1dfac	[#2057] meta: Fix concurrent mode changes ... Includes: 1. mode change read lock operation in every exported method that r/w the underlying database; 2. returning `ErrDegradedMode` logical error if any exported method is called in degraded (without a metabase) mode. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	3d6defd3e8	[#2057] meta: Do not lock the whole meta on GET ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	fa231b8c56	[#2057] blobstor: Block operations on a mode change ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	b673d9e472	[#2053] engine: Do not switch mode because of logical errors ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Anton Nikiforov	9a20498f34	[#1940] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2022-11-19 11:01:04 +03:00
Pavel Karpy	634792077e	[#1502] node: Store lock object on every container node ... Includes extending listing methods in the Storage Engine with object types. It allows tuning replication/policer algorithms: container nodes do not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement on the ohter container nodes. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	3b61cb4f49	[#1502] engine: Check all shards for LOCK'ing before inhuming ... It allows keeping all the locked objects safe after metabase resynchronization. Currently, all `LOCK` objects are broadcast to all nodes in a container, it guarantees `LOCK` object presence in a regular situation. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	34e8d2ba56	[#1502] shard: Add IsLocked method ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	9a039ba582	[#1502] meta: Add IsLocked method ... It gets an object and returns its locking status. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	d65604ad30	[#1985] blobstor: Allow to report multiple errors to caller ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	f2d7e65e39	[#2035] engine: Allow moving to degraded from background workers ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	b0e94b6a6b	[#1906] writecache: Do not require read-only mode in Flush ... It was needed before we started to flush during transition to `degraded` mode. Now it is confusing. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Pavel Karpy	2849e465f9	[#1699] meta: Do not return SplitInfoError on Delete ... It is not an error: removing virtual object is expected and should be just skipped. Getting a virtual object with `raw` flag is considered as an impossible action, all the virtual objects removals will be handled via their children's removals implicitly. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	a3e7365cbd	[#1732] pilorama: Fill parent mark correctly ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	134f2ba02e	[#1732] pilorama: Fix backwards log insertion ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	2ef38cfbc4	[#1996] engine: Ignore pilorama.ErrTreeNotFound for write operations ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-19 11:01:04 +03:00
Evgenii Stratonikov	d8d3588e1b	[#1996] engine: Always select proper shard for a tree ... Currently there is a possibility for modifying operations to fail because of I/O errors and a new tree to be created on another shard. This commit adds existence check for modifying operations. Read operations remain as they are, not to slow things. `TreeDrop` is an exception, because this is a tree removal and trying multiple shards is not an unwanted behaviour. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-03 15:29:23 +03:00
Evgenii Stratonikov	777fd32d4f	[#1818] writecache: Increase error counter on background errors ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	bffb0f894c	[#1818] writecache: Update storage ID during flush ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	5cf75404dc	[#1818] metabase: Add UpdateStorageID operation ... By default writecache puts the whole object to update storage ID. This logic comes from the times when we needed to put objects in the metabase by the writecache itself. Now this is done by the blobstor at unmarshaling objects during flush only to update storage ID is an overkill. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	b64b14eb54	[#1818] writecache: Reuse FSTree flushing code between flushes ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	a56927e3d4	[#1818] writecache: Remove unused variable ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-02 14:24:02 +03:00
Evgenii Stratonikov	98a152256b	[#1992] writecache: Allow to open in NOSYNC mode ... Applicable only to FSTree as we cannot handle corrupted databases properly yet. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-01 09:42:26 +03:00
Evgenii Stratonikov	f564430b90	[#1992] fstree: Allow working in SYNC mode ... Make O_SYNC the default and allow to opt-out explicitly. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-11-01 09:42:26 +03:00
Evgenii Stratonikov	34501685b7	[#1969] local_object_storage: Move ErrObjectIsExpired to another package ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-31 11:41:24 +03:00
Evgenii Stratonikov	56de2f1363	[#1969] local_object_storage: Simplify logic error construction ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-31 11:41:24 +03:00
Evgenii Stratonikov	fcdbf5e509	[#1969] local_object_storage: Add a type for logical errors ... All logic errors are wrapped in `logicerr.Logical` type and do not affect shard error counter. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-31 11:41:24 +03:00
Leonard Lyubich	e8c5f03c30	[#1905] shard: Don't log read-only errors of write-cache ... There is no need to log `writecache.ErrReadOnly` errors in `Delete` method of the `Shard`. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-28 18:30:45 +03:00
Leonard Lyubich	b1fa084756	[#1905] shard: Decrease severity level of write-cache failure logs ... In previous implementation `Shard.Delete` logged writecache's removal failures in `error` level. There is a need to decrease severity of these log records since they aren't critical and don't require individual review. Change level of the message to `info`. Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>	2022-10-28 18:30:45 +03:00
Evgenii Stratonikov	1e6588e761	[#1974] shard: Do not panic in degraded mode ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 12:41:12 +03:00
Evgenii Stratonikov	7395ab8ef7	[#1944] metabase: Assume static buckets are created on Init ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 12:41:12 +03:00
Evgenii Stratonikov	e4bc3d0e9d	[#1944] metabase: Recreate static buckets instead of resetting ... From the `Bucket.ForEach` doc: ``` The provided function must not modify the bucket; this will result in undefined behavior. ``` Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 12:41:12 +03:00
Evgenii Stratonikov	3b939d190c	[#1957] engine: Move shard to read-only if cannot move to degraded ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-26 08:20:53 +03:00
Evgenii Stratonikov	713fdab177	[#1907] shard: Return from Close after GC has stopped ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-25 11:54:45 +03:00
Pavel Karpy	41f93dcc1d	[#1953] blobstor: Log any object removal operation ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-24 21:52:18 +03:00
Pavel Karpy	942cc38146	[#1953] engine: Add refactored storage helpers ... After the refactor there are new storage characteristics: a type and a general storage id (that could be stringified). Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-24 21:52:18 +03:00
Pavel Karpy	0371d15b2f	[#1953] shard: Fix debug log messages ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-24 21:52:18 +03:00
Pavel Karpy	f8180447a1	[#1938] meta: Make version error messages more descriptive ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-24 21:46:18 +03:00
Evgenii Stratonikov	1beafea0b5	[#1869] shard: Add logs for SetMode operations on reload ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:26 +03:00
Evgenii Stratonikov	87be4f1629	[#1869] shard: Restore shard mode on failed reloads ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:26 +03:00
Evgenii Stratonikov	c785e11b20	[#1869] shard: Allow to reload metabase on SIGHUP ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:26 +03:00
Evgenii Stratonikov	f769fc83fc	[#1869] shard: Embed gcCfg as raw struct ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-24 13:28:26 +03:00
Pavel Karpy	49c38d642d	[#1902] engine: Search for the tree IDs in every shard ... Iterate over every shard and search for the container's trees. Final result is a concatenation of shards' results. It is considered that one fixed tree is placed on one fixed shard but the different trees of a fixed container could be placed on different shards. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	24e9e3f3bf	[#1902] engine, shard: Implement TreeList method ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	2e199c7ab1	[#1902] shard: Fix pilorama disabled err message ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Pavel Karpy	19850ef157	[#1902] pilorama: Add TreeList method ... To both `bolt` and `memory` forests; extend `Forest` interface. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-20 16:17:57 +03:00
Evgenii Stratonikov	1cb892c579	[#1910] .golangci.yml: Add misspell linker ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-18 15:08:26 +03:00
Evgenii Stratonikov	d772e35aba	[#1910] .golangci.yml: Add godot linker ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-18 15:08:26 +03:00
Pavel Karpy	f76516a883	[#1860] meta: Fix 0,1 -> 2+ version migration ... In the 2nd version, there was a database format change: buckets have changed their keys, so it becomes impossible to check the version in the 1 -> 2+ migrations because of different buckets that store info about the version. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-17 13:42:05 +03:00
Pavel Karpy	31c623636d	[#1863] node: Fix shard id in the object counter metrics ... If shard ID is stored in metabase (it is not the first time boot), read it, set it, use it (not a generated one) in the metrics writer. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-13 13:06:41 +03:00
Pavel Karpy	f037022a7a	[#1770] logger: Refactor Logger component ... Make it store its internal `zap.Logger`'s level. Also, make all the components to accept internal `logger.Logger` instead of `zap.Logger`; it will simplify future refactor. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-12 18:11:05 +03:00
Evgenii Stratonikov	19c0a74e94	[#1867] services/control: Allow to provide multiple shard IDs to some commands ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-12 11:20:48 +03:00
Evgenii Stratonikov	2d43892fc9	[#1840] neofs-node: Use blobstor paths to identify shard ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	4b005d3178	[#1840] blobstor: Return info about all components ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	6d7ffefec5	[#1840] blobstor/test: Add tests for auxiliary functions ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	328691c94f	[#1840] blobstor: Return Path from components ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	9b241e4a17	[#1840] neofs-node: Allow to use mode: disabled in config ... Currently, when removing shard special care must be taken with respect to shard numbering. `mode: disabled` allows to leave shard configuration in place while also ignoring it during initialization. This makes disk replacement much more convenient. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-10 11:14:55 +03:00
Evgenii Stratonikov	b2aa9947c2	[#1829] engine: Delete split objects properly ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-07 16:35:46 +03:00
Evgenii Stratonikov	6557f5d249	[#1839] engine: Handle Inhume errors properly ... If shard is in read-only or degraded mode, there is no need to increase error counter. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-07 14:25:52 +03:00
Pavel Karpy	4eb0ed11f8	[#1809] node: Do not boot up if metabase is outdated ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 12:32:10 +03:00
Evgenii Stratonikov	8b3b16fe62	[#1825] writecache: Flush cache when moving to the DEGRADED mode ... Degraded mode allows us to operate without an SSD, thus writecache should be unavailable in this mode. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 12:13:09 +03:00
Evgenii Stratonikov	2e3ef817f4	[#1819] engine: Increase error counter for PUT errors ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 10:11:52 +03:00
Evgenii Stratonikov	b89e71fa78	[#1819] common: Add ErrNoSpace ... Add a common error for this case because it is not an error which should increase error counter. Single error simplifies checks on the call-site. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 10:11:52 +03:00
Evgenii Stratonikov	af56574849	[#1819] engine: Fix error counter in Inhume ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-10-04 10:11:52 +03:00
Pavel Karpy	8ebe95747e	[#1770] node: Do not lock on shard's Close call ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Pavel Karpy	887afeaddb	[#1770] engine: Do not lock on shard init ... Init can take a lot of time. Because the mutex is taken, all new operations are blocked. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Pavel Karpy	fbd5bc8c38	[#1770] engine: Support configuration reload ... Currently, it only supports changing the compound of the shards. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Pavel Karpy	5c69e19016	[#1770] fstree: Depth parameter int -> uint64 ... Negative values have no sense. On the other hand it differs from the blobovnicza's configuration and prevents unification. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Pavel Karpy	2d7166f8d0	[#1770] shard: Move NewEpoch event routing on SE level ... It will allow dynamic shard management. Closing a shard does not allow removing event handlers. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-10-04 10:08:55 +03:00
Evgenii Stratonikov	0a411908ee	[#1806] writecache: Allow to ignore read errors during flush ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	f2045c10d7	[#1806] shard: Check each component mode when setting mode ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	3d882e9f47	[#1806] engine: Allow to flush write-cache ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	0b4c867ef1	[#1806] writecache: Allow to start flush manually ... Allow user to initiate flushing objects from a writecache. We need this in 2 cases: 1. During writecache storage schema update, it should be flushed with the old version of node and started clean with a new one. 2. During SSD replacement, to avoid data loss. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	55148404ee	[#1806] writecache: Use interfaces for blobstor and metabase ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-28 09:28:01 +03:00
Evgenii Stratonikov	6f45cc81fc	[#1764] neofs-node: Use constants for storage types ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-27 08:57:20 +03:00
Evgenii Stratonikov	a49137349b	[#1731] engine: Allow to use user handler for evacuated objects ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-24 13:47:48 +03:00
Evgenii Stratonikov	7cc68cf4d4	[#1810] blobovniczatree: Add a test for blobovnicva usage ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-23 10:19:26 +04:00
Evgenii Stratonikov	3df98ce7ba	[#1731] engine: Return the amount of actually moved objects in Evacuate ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-19 11:33:52 +03:00
Evgenii Stratonikov	a51b76056e	[#1731] engine: Add Evacuate command ... Make it possible to move all data from 1 shard to other shards. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-19 11:33:52 +03:00
Evgenii Stratonikov	7377979e12	[#1731] engine: Move single shard PUT to a separate function ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-19 11:33:52 +03:00
Evgenii Stratonikov	5321f8ef9c	[#1786] engine: Unify parameter setters ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-15 10:28:48 +03:00
Evgenii Stratonikov	b064fb24d8	[#1616] engine: Do not use batches in delete ... Use a simple loop at the callsite. This way we remove as much as we can. Also, `Delete` metrics is more meaningful now. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-15 10:28:48 +03:00
Evgenii Stratonikov	ae1dab29bc	[#1482] metabase: Encode database keys in binary ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-15 10:28:26 +03:00
Pavel Karpy	532c41ca05	[#1658] meta: Do not check object expiration in counters ... Do not perform operations that produce unused results in `syncCounter`. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	431e331373	[#1658] shard: Update metric counters ... Use meta's operation results to change the metrics. Support typed object counters. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	ad47e2a985	[#1658] meta: Add logic counter ... - Meta now supports (and requires) inc/dec labeled counters - The new logic counter is incremented on `Put` operations and is decremented on `Inhume` and `Delete` operations that are performed on _stored_ objects only - Allow force counters sync. "Force" mode should be used on metabase resync and should not be used on a regular meta start Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	d872862710	[#1658] meta: Force counters resync process ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	beb7f2a048	[#1634] node: Change default epoch in tests ... Do not treat objects with expiration as expired by default. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	edef26a4fd	[#1658] engine: Update metrics interfaces ... It now supports typed object counter metrics. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-13 21:32:37 +04:00
Pavel Karpy	4afb928ab6	[#1674] *: Expire entities after the expiration epoch ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-09-12 19:56:59 +04:00
Evgenii Stratonikov	60b2930417	[#1778] neofs-node: Fix blobovnicza config type ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-12 17:06:00 +03:00
Evgenii Stratonikov	87bd49563e	[#1761] pilorama: Use batch size of 1 for tests ... Improve tests speed by a lot and use more iterations in `ApplyRandom` test. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-12 09:54:15 +03:00
Evgenii Stratonikov	a2bb3a2a96	[#1630] pilorama: Support dropping trees ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-12 09:54:15 +03:00
Leonard Lyubich	8e8265d5ea	[#1707] blobovnicza/get: Iterate over all buckets regardless of config ... Re-configuration of Blobovnicza's object size limit must not affect already stored objects. In previous implementation `Blobovnicza` didn't see objects stored in buckets which became too big after size limit re-configuration. For example, lets consider 1st configuration with 64KB size limit for stored objects. Lets assume that we stored object of 64KB size, and re-configured `Blobovnicza` with 32KB limit. After reboot object should be still available, but actually it isn't. This is caused by `Get` operation algorithm which iterates over configured size ranges only, and doesn't process any other existing size bucket. By the way, increasing of the object size limit didn't lead to the problem even in previous implementation. Make `Blobovnicza.Get` method to iterate over all size buckets regardless of current configuration. This covers the described scenario. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>	2022-09-08 12:24:28 +04:00
Evgenii Stratonikov	e6cf0b3d42	[#1745] writecache: Set flush parameters based on max size ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-02 11:59:53 +03:00
Evgenii Stratonikov	c04126f35e	[#1745] writecache: Remove IsErrNotFound ... We specify the error in the doc-comment, and it is the same for all our components. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-02 11:59:53 +03:00
Evgenii Stratonikov	9e41e85295	[#1745] writecache: Actualize docs ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-02 11:59:53 +03:00
Evgenii Stratonikov	e9c6ee2623	[#1745] writecache: Rename constants for default values ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-02 11:59:53 +03:00
Evgenii Stratonikov	177e8e01b1	[#1745] neofs-node: Remove memcache_capacity from the configuration ... It is unused since ddaed283e9 . Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-02 11:59:53 +03:00
Evgenii Stratonikov	bda084f331	[#1745] writecache: Simplify object counters ... Remove unused option and additional pointers to db/fstree. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-02 11:59:53 +03:00
Evgenii Stratonikov	20abdaeed4	[#1745] writecache: Set flush mark only on success ... Set flush mark in the inside the flush worker because writing to the blobstor can fail. Because each evicted object must be deleted, it is reasonable to do this in the evict callback. The evict callback is protected by LRU mutex and thus potentially interferes with `Get` and `Iterate` methods. This problem will be addressed in the future. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-09-02 11:59:53 +03:00
Evgenii Stratonikov	d8a00c365a	[#1691] blobovniczatree: Fix active blobovnicza caching ... Maintain an invariant that any blobovnicza is present either in `opened` or in `active` map. Otherwise, the logic becomes too complicate because it is not obvious when we should close the blobovnicza. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-31 19:11:48 +03:00
Evgenii Stratonikov	dfac4e1c0b	[#1691] blobovniczatree: Do not fill cache during initialization ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-31 19:11:48 +03:00
Evgenii Stratonikov	ae52d53609	[#1698] pilorama: Add a test for the empty FileName ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 16:52:51 +03:00
Evgenii Stratonikov	78bf17c0b0	[#1686] blobstor/*: Remove unneeded tests ... The blobovniczatree test also checks the amount of objects we can put, so leave it here. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	b9a2055e1c	[#1686] blobstor/*: Return ErrReadOnly for modifying operations ... This check should occur on the shard level, but because blobstor components expose `Open(readOnly bool)` interface, it is reasonable to expect an error here. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	6c2d3b020f	[#1686] fstree: Return proper error for concurrent Delete ... If the file doesn't exist, return `apistatus.ObjectNotFound`. First check is still there as a shortcut. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	0b95a21701	[#1686] blobstor: Add generic tests ... This tests check that each blobstor component behaves similarly when same methods are being used. It is intended to serve as a specification for all future components. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	b2d4cc556e	[#1686] blobovnicza: Remove IterateObjects ... It is used only once, makes sense to inline. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	5f95498364	[#1686] blobstor: Add generic tests ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	9792e6a4ef	[#1686] metabase: Add generic tests ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	482a7e7f2f	[#1686] pilorama: Add generic tests ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	b8b9d25f9d	[#1686] local_object_storage: Add generic tests ... Use them for writecache as a simple example. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Evgenii Stratonikov	bc20851cd1	[#1686] shard: Open all components in case metabase has failed ... All components must be opened first, before any other operation is performed. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-30 10:02:43 +03:00
Pavel Karpy	c7c1c257e1	[#1658] node: Init object counter on meta's Init ... Includes: 1. Renaming counter key to distinguish logical and physical objects 2. Version update dropping since changes could be done in a compatible way Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-25 19:20:33 +03:00
Pavel Karpy	9c7b3ce799	[#1658] node: Read metrics from meta on startup ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-25 19:20:33 +03:00
Pavel Karpy	745f72fff0	[#1658] node: Support object counter metrics ... Increment shard's object counter on successful `Put` calls and decrement on `Delete`. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-25 19:20:33 +03:00
Pavel Karpy	626766db08	[#1658] meta: Add object counter ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-25 19:20:33 +03:00
Pavel Karpy	30341f2192	[#1687] *: Perform go fmt using go v1.19 ... Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-22 18:59:57 +03:00
Evgenii Stratonikov	e8c6dce466	[#1523] blobstor: Rename compression.CConfig to Config ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	0042c1d4a6	[#1584] blobovniczatree: move public methods to different files ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	26b4a258e0	[#1523] neofs-node: Refactor configuration ... 1. Move compression parameters to the `shard` section. 2. Allow to use multiple sub-storage components in the blobstor. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	13cdbde2e2	[#1523] blobstor: Remove unused types ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	0c9e4e6a35	[#1523] blobstor: Unify request dispatch logic ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	266458fe5c	[#1523] blobstor: Unify fstree and blobovnicza interfaces ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	5aa3defc67	[#1523] writecache: Remove cloneBytes ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	ddaed283e9	[#1523] writecache: Simplify logic ... 1. Remove in-memory cache. It doesn't persist objects and if we want more speed, `NoSync` option can be used for the bolt DB. 2. Put to the metabase in a synchronous fashion. This considerably simplifies overall logic and plays nicely with the metabase bolt DB batch settings. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	4176b2a1bc	[#1523] local_object_storage: Unify parameters for the Iterate operation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	73f8bb3e5f	[#1523] shard: Store generic storage ID in metabase ... Allow to extend blobstor with more storage sub-systems. Currently objects stored in the FSTree have empty byte slice descriptor and object from blobovnicza tree have the same id as earlier. Each such change in the identifier formation should be accompanied with metabase version increase. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	9eb018672c	[#1523] blobstor: Unify parameters for Put operation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	7d6df543d7	[#1523] blobstor: Export single Get, GetRange and Delete methods ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	ca15083a50	[#1523] local_object_storage: Unify parameters for the Exists operation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	d75d030a90	[#1523] local_object_storage: Unify parameters for the Delete operation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	36c88f0dc8	[#1523] local_object_storage: Unify parameters for the GetRange operation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	d9a2f280c9	[#1523] local_object_storage: Unify parameters for the Get operation ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	b621f5983a	[#1523] local_object_storage: Move blobovnicza tree to a separate package ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-08-22 13:14:19 +03:00
Evgenii Stratonikov	4e361e2ab4	[#1647] writecache: Open DB in sync mode ... Close #1647. Initially, `Sync: false` was provided because we can already lose objects cached in memory. However, future changes in writecache will remove inmemory cache and speed up it via other means. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-08-16 15:27:18 +04:00
Pavel Karpy	da2975a2f9	[#1664] write-cache: Fix panic on Delete operation ... If an object is found in the Write-cache and is placed at the end of the in-memory cache, the memory counter update operation tries to dereference the index that is out of the sliced array. Moreover, even if panic does not appear, the counter is updated with the wrong value. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-09 20:18:26 +03:00
Pavel Karpy	a5da665e69	[#1656] meta: Fix MatchCommonPrefix operation ... Return all the objects on the empty common prefix search without search optimizations that breaks boltDB logic. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-04 16:32:00 +03:00
Pavel Karpy	156ba85326	[#1634] node: Do not return expired objects ... If an object has not been marked for removal by the GC in the current epoch yet but has already expired, respond with `ErrObjectNotFound` api status. Also, optimize shard iteration: a node must stop any iteration if the object is found but gonna be removed soon. All the checks are performed by the Metabase. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-04 16:31:49 +03:00
Pavel Karpy	9aba0ba512	[#1634] meta: Add epoch state ... It allows performing expiration checks on the stored objects. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>	2022-08-04 16:31:49 +03:00
Evgenii Stratonikov	8ffc2fdf5e	[#1642] engine: Do not increase error counter if the pilorama is disabled ... After a4adb79db new logical error could be returned. Do not increase error counter in this case. Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-07-30 18:39:22 +03:00
Evgenii Stratonikov	8d0884e74f	[#1644] storagelog: Fix doc comment ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-07-30 18:39:08 +03:00
Evgenii Stratonikov	4abe5a7245	shard: add more checks for GetRange parameters ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-07-29 11:22:04 +03:00
Evgenii Stratonikov	72586f17d4	shard: fix GetRange for objects stored in the write-cache ... Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>	2022-07-29 11:22:04 +03:00
Evgenii Stratonikov	1691364653	[#1559] local_object_storage: Fix tests and some data races ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	50e28f22f9	[#1559] shard: Change Degraded mode string representation ... It is a flag, but is a `degraded-read-write` mode for a user. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Leonard Lyubich	fabe717d32	[#1549] shard: Turn to read-only mode on metabase failure ... If metabase can't be opened in the default mode, try opening shard first in `ReadOnly` mode and then in `DegradedReadOnly`. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	4944490ffb	[#1559] local_object_storage: Move shard to the DegradedReadOnly mode ... `Degraded` mode can be set by the administrator if needed. Modifying operations in this mode can lead node into an inconsistent state because metabase checks such as lock checking are not performed. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	3df62769c0	[#1559] local_object_storage: Allow to set mode for all components ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	1e786233bf	[#1559] local_object_storage: Provide readOnly flag to Open ... We should be able to reopen storage in readonly in runtime. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Leonard Lyubich	e38b0aa4ba	[#1549] engine: Disable shard on blobovnicza init failure ... There is a need to support working w/o shard if it has problems with blobovnicza tree. Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove shard from storage engine's shard set if it returned this error from `Init` call. So if some of the shards (but not all) return this error, the node will be able to continue working without them. Signed-off-by: Leonard Lyubich <leonard@nspcc.ru> Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	40a56c6b42	[#1559] engine: Do not count logical errors as storage ones ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	c8911d72d0	[#1559] shard: Do not consult metabase in a degraded mode ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	339864b720	[#1559] local_object_storage: Move shard.Mode to a separate package ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	d8ba954aff	[#1559] shard: Use Set prefix for parameter setting ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	7b882b26d8	[#1559] shard: Remove public functions ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	34d319fed2	[#1559] metabase: Use Set prefix for parameter setting ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	f58234aa2f	[#1559] metabase: Remove public functions ... Reduce public interface of this package. Later each result will contain an additional status, so it makes more sense to use the same functions and result processing everywhere. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 17:56:06 +03:00
Evgenii Stratonikov	a4adb79db7	[#1607] pilorama: Enable tree service explicitly ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00
Evgenii Stratonikov	d62723f038	[#1505] pilorama: Provide timeout to bbolt.Open ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00
Evgenii Stratonikov	26041f18bf	[#1505] pilorama: Allow to customize database parameters ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00
Evgenii Stratonikov	c7b10598f9	[#1333] engine: Increase error counter for pilorama errors ... 1. Modifying operations are not expected to fail, unless the shard is read-only. 2. `Get*` operations should increase error counter too, unless the error is `ErrTreeNotFound`. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00
Evgenii Stratonikov	5e843a73f9	[#1333] services/control: Return pilorama info in ListShards RPC ... Do not return backend type from the service for now, because memory backend is expected to vanish. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00
Evgenii Stratonikov	8f4ee1aded	[#1333] local_object_storage: Support ReadOnly mode in pilorama ... The tricky part here is the engine itself: we stop iteration on `ErrReadOnly` because it is better to synchronize the shard later than to have partial trees stored in 2 shards. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00
Evgenii Stratonikov	735931c842	[#1481] pilorama: Fix TreeApply ... Current implementation prevents invalid operations to become valid at some later point (consider adding a child to the non-existent parent and then adding the parent). This seems to diverge from the paper algorithm and complicates implementation. Make it simpler. Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00
Evgenii Stratonikov	ad3038d16d	[#1444] pilorama: Fix TreeMove in bbolt backend ... Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00
Evgenii Stratonikov	8027b7bb6b	[#1444] pilorama: Optimize internal encoding/decoding ... ``` name old time/op new time/op delta ApplySequential/bbolt-8 55.5µs ± 4% 55.5µs ± 3% ~ (p=1.000 n=10+7) ApplyReorderLast/bbolt-8 108µs ± 6% 112µs ± 8% ~ (p=0.077 n=9+9) name old alloc/op new alloc/op delta ApplySequential/bbolt-8 28.8kB ± 3% 27.7kB ± 6% -3.79% (p=0.005 n=10+10) ApplyReorderLast/bbolt-8 41.4kB ± 5% 38.9kB ± 5% -6.19% (p=0.001 n=10+9) name old allocs/op new allocs/op delta ApplySequential/bbolt-8 262 ± 2% 235 ±10% -10.41% (p=0.000 n=10+10) ApplyReorderLast/bbolt-8 684 ± 6% 616 ± 7% -10.04% (p=0.000 n=10+9) ``` Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>	2022-07-21 15:08:24 +03:00