1064 commits

Author	SHA1	Message	Date
Evgenii Stratonikov	e5e0542482	[#1085] log: Move storage log message to constants package ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-15 07:57:00 +00:00
Evgenii Stratonikov	5be36924e3	[#41] log: Log storage operations in only in Debug ... They are mostly useless unless we need to _debug_ a specific issue. The amount of logs we produce is too big. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-15 07:57:00 +00:00
Dmitrii Stepanov	40781b3a20	[#1086] engine: Change mode in case of errors async ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-10 12:29:43 +00:00
Evgenii Stratonikov	5ef5734c4e	Reapply "[#972] Drop x/exp/slices dependency" ... This reverts commit 946f2ec2bf. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-10 12:09:34 +00:00
Evgenii Stratonikov	2c4b50a71e	Reapply "[#972] Use require.ElementsMatch() where possible" ... This reverts commit 7627d08914. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-10 12:09:34 +00:00
Evgenii Stratonikov	3dc81cb4fc	Reapply "[#972] Use min/max builtins" ... This reverts commit dad56d2e98. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-10 12:09:34 +00:00
Evgenii Stratonikov	d864945961	Reapply "[#972] pilorama: Remove removeDuplicatesInPlace()" ... This reverts commit 9f68305c2e. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-10 12:09:34 +00:00
Dmitrii Stepanov	1b17258c04	[#1029] metabase: Add refill metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-10 13:05:44 +03:00
Dmitrii Stepanov	e3d9dd6ee8	[#1024] blobovnicza: Copy data on iterate ... DB value is only valid while the tx is alive. But handler may to run something in other goroutine. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-10 10:21:11 +03:00
Dmitrii Stepanov	57466594fb	[#1024] shard: Resync metabase concurrently ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-10 10:21:10 +03:00
Dmitrii Stepanov	1005bf4f56	[#1024] shard: Add refill metabase benchmark ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-10 10:21:10 +03:00
Dmitrii Stepanov	76398c06b0	[#1080] metabase: Add StorageID metric ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-10 10:00:08 +03:00
Dmitrii Stepanov	7b1adfed3e	[#1080] metabase: Open bucket for container counter once ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-10 10:00:08 +03:00
Dmitrii Stepanov	5b8200de88	[#984] blobovnicza: Do not fail rebuild on big objects ... If blobovnicza contains objects larger than object size parameter value, then rebuild fails with an error, because there is no such bucket in database. This commit forces to create bucket on rebuild. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-09 11:51:18 +00:00
Anton Nikiforov	d614f04a0a	[#1072] Fix gofumpt issues ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-04-03 22:21:14 +03:00
Evgenii Stratonikov	17af91619a	[#1070] pilorama: Fix cycling behaviour for sorted listing ... In case there are no items left, return empty slice. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-02 14:41:31 +00:00
Ekaterina Lebedeva	d5194ab2a6	[#949] metabase: fix shard.UpdateID() ... metabase.Open() now reports metabase mode metric. shard.UpdateID() needs to read shard ID from metabase => needs to open metabase. It caused reporting 'shard undefined' metrics. To avoid reporting wrong metrics metabase.GetShardID() was added which also opens metabase and does not report metrics. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-04-01 17:27:34 +03:00
Ekaterina Lebedeva	81a0346a96	[#949] metabase: fix metabase mode metric ... It used to always show CLOSED regardless of actual mode. Now metric represents actual metabase mode of operations. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-04-01 17:27:34 +03:00
Evgenii Stratonikov	e12fcc041d	[#1059] services/tree: Fast sorted listing ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-01 12:37:34 +00:00
Evgenii Stratonikov	f23e38c285	Revert "[#446] engine: Move to read-only on blobstor errors" ... This reverts commit 69df0d21c2. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-04-01 12:48:30 +03:00
Dmitrii Stepanov	942d83611b	[#874] engine: Revert Check object existance concurrently ... This reverts commit f526f49995. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-04-01 08:42:34 +00:00
Ekaterina Lebedeva	0990a9b0bd	[#1055] blobstor: fix mode metric ... It used to always show CLOSED after setting shard mode to read-only regardless of actual mode. Now metric represents actual blobstor mode of operations. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-03-29 20:44:47 +00:00
Dmitrii Stepanov	c09c701613	[#1048] metabase: Fix drop buckets during resync ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-03-19 14:28:31 +03:00
Dmitrii Stepanov	31e2396a5f	[#1043] control: Add ResetEvacuationStatus implementation ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-03-13 10:29:45 +00:00
Anton Nikiforov	3195142d67	[#959] writecache: Avoid manipulation with cache in DEGRADED mode ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-03-11 18:35:41 +00:00
Dmitrii Stepanov	d433b49265	[#973] node: Resolve perfsprint linter ... `fmt.Errorf can be replaced with errors.New` and `fmt.Sprintf can be replaced with string addition` Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-03-11 17:55:50 +03:00
Dmitrii Stepanov	66a26b7775	[#973] node: Resolve revive: unused-parameter linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-03-11 17:11:49 +03:00
Dmitrii Stepanov	0882840bf5	[#634] shard: Add writecache inhume tests ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-03-06 13:12:34 +03:00
Anton Nikiforov	ae5bb87e70	Revert "[#866] Use TTL for blobovnicza tree cache" ... This reverts commit d9cbb16bd3. Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-03-01 19:29:33 +03:00
Dmitrii Stepanov	d6534fd755	[#1016] frostfs-node: Fix gopls issues ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-03-01 12:13:43 +03:00
Dmitrii Stepanov	918613546f	[#1008] metabase: Do not update storageID on put ... There may be a race condition between put an object and flushing the writecache: 1. Put object to the writecache 2. Writecache flushes object to the blobstore and sets blobstore's storageID 3. Put object to the metabase, set writecache's storageID Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-02-28 11:01:50 +03:00
Dmitrii Stepanov	2ad433dbcb	[#1005] engine: Drop shards weights ... Unused. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-02-26 17:25:05 +03:00
Dmitrii Stepanov	7470c383dd	[#997] metabase: Drop toMoveIt bucket ... It is not used. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-02-21 10:06:05 +03:00
Evgenii Stratonikov	adf7ebab5b	[#996] metabase: Speed up bucket creation ... Most of the time it exits, e.g. when it is per-container and use on each object PUT. Bbolt implementation first tries to create bucket and then returns it if it exists. Create operation uses cursor and thus is not very lightweight, we can avoid it. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ new │ │ sec/op │ sec/op vs base │ Put/parallel-8 174.4µ ± 3% 163.3µ ± 3% -6.39% (p=0.000 n=10) Put/sequential-8 263.3µ ± 2% 259.0µ ± 1% -1.64% (p=0.000 n=10) geomean 214.3µ 205.6µ -4.05% │ old │ new │ │ B/op │ B/op vs base │ Put/parallel-8 275.3Ki ± 3% 281.1Ki ± 4% ~ (p=0.063 n=10) Put/sequential-8 413.0Ki ± 2% 426.6Ki ± 2% +3.29% (p=0.003 n=10) geomean 337.2Ki 346.3Ki +2.70% │ old │ new │ │ allocs/op │ allocs/op vs base │ Put/parallel-8 678.0 ± 1% 524.5 ± 2% -22.64% (p=0.000 n=10) Put/sequential-8 1.329k ± 0% 1.183k ± 0% -10.91% (p=0.000 n=10) geomean 949.1 787.9 -16.98% ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-20 15:42:58 +00:00
Evgenii Stratonikov	9f68305c2e	Revert "[#972] pilorama: Remove removeDuplicatesInPlace()" ... This reverts commit 45fd4e4ff1. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 15:36:01 +00:00
Evgenii Stratonikov	dad56d2e98	Revert "[#972] Use min/max builtins" ... This reverts commit 89784b2e0a. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 15:36:01 +00:00
Evgenii Stratonikov	7627d08914	Revert "[#972] Use require.ElementsMatch() where possible" ... This reverts commit 6d9707ff1f. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 15:36:01 +00:00
Evgenii Stratonikov	946f2ec2bf	Revert "[#972] Drop x/exp/slices dependency" ... This reverts commit f3e50772fd. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 15:36:01 +00:00
Evgenii Stratonikov	f3e50772fd	[#972] Drop x/exp/slices dependency ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 13:13:09 +00:00
Evgenii Stratonikov	6d9707ff1f	[#972] Use require.ElementsMatch() where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 13:13:09 +00:00
Evgenii Stratonikov	89784b2e0a	[#972] Use min/max builtins ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 13:13:09 +00:00
Evgenii Stratonikov	45fd4e4ff1	[#972] pilorama: Remove removeDuplicatesInPlace() ... Also, check that slices.CompareFunc() indeed passes all the tests before removal. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-19 13:13:09 +00:00
Anton Nikiforov	05b5f5ca85	[#959] writecache: Fix panic on Get when it is not initialized ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-14 16:10:33 +03:00
Anton Nikiforov	2429508ac5	[#959] shard: Skip rebuild in DEGRADED mode ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-14 15:39:28 +03:00
Anton Nikiforov	0bd030507e	[#948] metrics: Set actual value for shard_id after restart ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:43:21 +03:00
Anton Nikiforov	6a5769d1da	[#948] Fix gofumpt issue ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-02-13 09:40:46 +03:00
Evgenii Stratonikov	b36a453238	[#970] fstree: Add build tag to enable generic version on linux ... Unless tested, generic version can start gaining bugs. With a separate build tag we can have the best of both worlds: 1. Use optimized implementation for linux by default. 2. Run tests or benchmarks for both. Note that they are not actually run automatically now, but this is at leas possible. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Evgenii Stratonikov	abd502215f	[#970] fstree: Move file locking to the generic writer ... It is not a part of FSTree itself, but rather a way to solve concurrent counter update on non-linux implementations. New linux implementations is pretty simple: link fails when the file exists, unlink fails when the file doesn't exist. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Evgenii Stratonikov	fb74524ac7	[#970] fstree: Move delete implementation to a separate file ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00
Evgenii Stratonikov	7f692409cf	[#970] fstree: Handle unsupported O_TMPFILE ... Metabase test relied on this behaviour, so fix the test too. Cherry-picking was hard and did too many conflicts, here is an original PR: https://github.com/nspcc-dev/neofs-node/pull/2624 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-02-09 16:12:11 +00:00