353 commits

Author	SHA1	Message	Date
Anton Nikiforov	e3573de6db	[#930] gc: Stop internal activity by context ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-01-31 08:30:34 +00:00
Dmitrii Stepanov	675eec91f3	[#938] shard: Update only changed counters ... If metric value hasn't changed, but we update metric, then non existed metric will apear with zero value. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-30 12:37:48 +03:00
Dmitrii Stepanov	931a5e9aaf	[#918] engine: Move shard to degraded mode if metabase open failed ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-23 11:16:40 +03:00
Dmitrii Stepanov	f526f49995	[#874] engine: Check object existance concurrently ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-23 09:28:29 +03:00
Dmitrii Stepanov	57171907e3	[#904] metabase: Return if object was actuall inserted ... This requires to count metrics properly. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-17 19:04:39 +03:00
Dmitrii Stepanov	a2ab373a0a	[#895] metabase: Do not delete GC mark for virtual objects ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-11 12:32:09 +00:00
Dmitrii Stepanov	47dcfa20f3	[#895] test: Use t.Cleanup only for external resources ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-11 12:32:09 +00:00
Dmitrii Stepanov	f1b2b8bffa	[#895] test: Fix NewLogger arguments list ... `debug` is always true. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-11 12:32:09 +00:00
Dmitrii Stepanov	4b8b4da681	[#864] engine: Drop container count metric if container removed ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-10 10:45:32 +03:00
Dmitrii Stepanov	d75e7e9a21	[#864] engine: Drop container size metric if container deleted ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-10 10:44:54 +03:00
Dmitrii Stepanov	dfd62ca6b1	[#864] metabase: Refactor delete/inhume ... Available -> Logic, Raw -> Phy for delete/inhume results. Use single counter instead of vectors. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-01-09 09:59:42 +03:00
Dmitrii Stepanov	581887148a	[#569] cli: Add control shards writecache seal command ... It does the same as `control shards flush-writecache --seal`, but has better name. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-29 16:05:37 +03:00
Dmitrii Stepanov	0cb0fc1735	[#569] writecache: Allow to seal writecache after flush ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-29 16:05:37 +03:00
Dmitrii Stepanov	8180a0664f	[#887] node: Drop badger writecache implementation ... Badger implementation isn't tested and works not well, but requires human resources to maintain. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-22 13:00:54 +03:00
Dmitrii Stepanov	3b7c0362a8	[#861] shard: Fix Delete object ... It is possible that object doesn't exist in metabase. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-12 14:25:40 +03:00
Dmitrii Stepanov	f1c7905263	[#661] blobovniczatree: Make Rebuild concurrent ... Different DBs can be rebuild concurrently. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-07 15:37:33 +03:00
Dmitrii Stepanov	422226da18	[#661] blobovniczatree: Add Rebuild implementation ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-07 15:37:32 +03:00
Dmitrii Stepanov	44552a849b	[#661] shard: Add blobstor rebuilder ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-07 15:37:32 +03:00
Dmitrii Stepanov	d30ab5f29e	[#838] metabase: Count user objects ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-06 15:44:21 +03:00
Dmitrii Stepanov	f314da4af3	[#838] metabase: Add user object type counter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-06 15:44:21 +03:00
Dmitrii Stepanov	29550fe600	[#838] shard: Refactor updateMetrics method ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-12-06 15:44:21 +03:00
Alexander Chuprov	445ebcc0e7	[#651] shard/test: Speedup Shard_Delete ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2023-11-30 13:19:43 +00:00
Alexander Chuprov	2302e5d342	[#651] shard/test: Refactor Shard_Delete ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2023-11-30 13:19:43 +00:00
Dmitrii Stepanov	07390ad4e3	[#715] node: Unify config parameter names ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-11-22 17:13:50 +03:00
Dmitrii Stepanov	76ff26039c	[#96] node: Drop neo-go's slices package ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-11-17 13:24:04 +03:00
Dmitrii Stepanov	70ab1ebd54	[#763] metrics: Add container_objects_total metric ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-11-08 12:30:57 +03:00
Dmitrii Stepanov	79088baa06	[#772] node: Apply gofumpt ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-10-31 17:03:03 +03:00
Evgenii Stratonikov	00aa6d9749	[#633] shard/test: Fix TestCounters() ... Introduced in 362f24953a, forgotten to be changed because test generator didn't provide payload size. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-10-31 12:53:28 +00:00
Evgenii Stratonikov	b8f79f4227	[#633] shard/test: Fix race conditions in TestCounters() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-10-31 12:53:28 +00:00
Dmitrii Stepanov	f2437f7ae9	[#734] shard: Fix Delete method ... Due to the flushing data from the writecache to the storage and simultaneous deletion, a partial deletion situation is possible. So as a solution, deletion is allowed only when the object is in storage, because object will be deleted from writecache by flush goroutine. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2023-10-16 17:00:18 +03:00
Alexander Chuprov	c1e4130020	[#146] node: Add trace_id to logs ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2023-09-27 11:05:27 +03:00
Ekaterina Lebedeva	8a81af5a3b	[#653] Add context parameter to Open functions ... Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2023-09-07 18:03:29 +03:00
Evgenii Stratonikov	a716db99db	[#668] shard/test: Do not alter rootPath option ... Supposedly, this was added to allow creating 2 different shards without subtest. Now we use t.TempDir() everywhere, so this should not be a problem. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	36759f8434	[#668] shard/test: Properly check event processing ... See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	39879fa868	[#668] shard/test: Add dontRelease options ... Most of the time we would like to close shard with minor exceptions. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	c661ba1312	[#668] shard/test: Use sane defaults in the test constructor ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	268adb79cb	[#668] shard/test: Simplify shard construction ... newCustomShard() has many parameters but only the first is obligatory. `enableWriteCache` is left as-is, because it directly affects the functionality. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	429f941cda	[#668] shard/test: Release shard in t.Cleanup() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	382eb8a485	[#668] shard/test: Disable GC where it is not needed ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	42696016de	[#668] shard: Close stopChannel in GC ... It is done once, but now we could read it from multiple places. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	bdecfbc1be	[#668] shard/test: Move tests to the main package ... Semantic patch (also, duplicate definitions are removed): ``` @@ var e identifier @@ -import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard" -shard.e +e ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	aa23c6a83a	[#668] shard/test: Remove subtest from TestCounters ... Otherwise, individual tests cannot be run. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	da8f384324	[#668] shard/test: Fix typo in existence ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-09-07 07:39:39 +00:00
Evgenii Stratonikov	b9b86d2ec8	[#666] shard/test: Fix data race in metrics tests ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-31 08:39:42 +00:00
Evgenii Stratonikov	9072772a09	[#649] shard/test: Increase GC remover interval ... This was set in #348 to speed up tests. It seems 100ms doesn't increase overall test time, but it reduces the amount of logs by 100x factor. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-25 10:10:25 +00:00
Evgenii Stratonikov	c4db8e7690	[#637] shard/test: Fix data race ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-25 10:10:25 +00:00
Evgenii Stratonikov	96e690883f	[#638] Unify test loggers ... In some places we have debug=false, in others debug=true. Let's be consistent. Semantic patch: ``` @@ @@ -test.NewLogger(..., false) +test.NewLogger(..., true) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-23 11:21:05 +00:00
Evgenii Stratonikov	322c1dc273	[#638] Use test.NewLogger() in tests ... Semantic patch (restricted to **/*_test.go): ``` @@ @@ +import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test" -import "go.uber.org/zap" -import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger" -&logger.Logger{Logger: zap.L()} +test.NewLogger(t, false) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-23 11:21:05 +00:00
Evgenii Stratonikov	82cc453be9	[#xx] shard: Fix data race in metrics tests ... Protect test metric store fields with a mutex. Probably, not every field should be protected, but better safe than sorry. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2023-08-23 10:26:12 +00:00
Alexander Chuprov	4d9a6c07fb	[#618] core: Replace fmt.Sprintf with strconv.FormatUint ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2023-08-21 07:14:50 +00:00