1186 commits

Author	SHA1	Message	Date
Dmitrii Stepanov	3da168f8cf	[#1412] shard: Resolve container is indexed on metabase resync ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:51 +03:00
Dmitrii Stepanov	4572fa4874	[#1412] searchSvc: Check container is indexed ... For non S3 containers it is expected to use attributes index for some attributes. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:50 +03:00
Dmitrii Stepanov	1efa64ee72	[#1412] metabase: Add search by indexed attributes ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:50 +03:00
Dmitrii Stepanov	be744ae3e6	[#1412] metabase: Index attributes for indexed containers ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:49 +03:00
Dmitrii Stepanov	1b520f7973	[#1412] engine: Add IsIndexedContainer flag ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:49 +03:00
Dmitrii Stepanov	899cd55c27	[#1412] engine: PutPrm refactoring ... Use fields instead of methods. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:48 +03:00
Evgenii Stratonikov	fc032838c0	[#1215] blobstor/test: Cover iteration behaviour ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-07 12:44:27 +00:00
Anton Nikiforov	2f710d8f94	[#1414] metabase: Check parameter for CountAliveObjectsInBucket ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-10-04 17:11:15 +03:00
Evgenii Stratonikov	4dc9a1b300	[#1413] engine: Remove error counting methods from Shard ... All error counting and hangling logic is present on the engine level. Currently, we pass engine metrics with shard ID metric to shard, then export 3 methods to manipulate these metrics. In this commits all methods are removed and error counter is tracked on the engine level exlusively. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-04 15:10:17 +03:00
Evgenii Stratonikov	963faa615a	[#1413] engine: Cleanup shard error reporting ... - `reportShardErrorBackground()` no longer differs from `reportShardError()`, reflect this in its name; - reuse common pieces of code to make it simpler. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-04 15:10:17 +03:00
Evgenii Stratonikov	9a87acb87a	[#1410] engine: Provide the default implementation to MetricsRegister ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-03 08:23:06 +00:00
Evgenii Stratonikov	9206ce5cd2	[#1410] shard: Provide the default implementation for MetricsWriter ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-03 08:23:06 +00:00
Evgenii Stratonikov	6c46044c9c	[#1410] shard: Move MetricsWriter interface to a separate file ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-03 08:23:06 +00:00
Dmitrii Stepanov	01e3944b31	[#1408] metabase: Fix tests ... No need to specify container ID for objects created with `testutil.GenerateObjectWithCID`. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-02 14:41:40 +03:00
Dmitrii Stepanov	434048e8d9	[#1408] metabase: Fix EC search with slow and fast filters ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-02 14:36:59 +03:00
Anton Nikiforov	d0ed29b3c7	[#1350] node: Add ability to evacuate objects from REP 1 only ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-09-27 15:41:17 +03:00
Dmitrii Stepanov	401c398704	[#1388] metabase: Drop unused ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-25 08:55:38 +00:00
Dmitrii Stepanov	004ff9e9bf	[#1388] blobstor: Drop unused ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-25 08:55:38 +00:00
Dmitrii Stepanov	63a567a1de	[#1388] engine: Drop unused ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-25 08:55:38 +00:00
Dmitrii Stepanov	fd18aa363b	[#1385] metabase: Optimize isTomb check ... As tombstone and target must have the same containerID, do not iterate other containers. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-24 17:09:06 +03:00
Dmitrii Stepanov	76268e3ea2	[#1385] metabase: Validate that tombstone and target have the same container ID ... Target container ID is taken from tombstone: cmd/frostfs-node/object.go:507 Also object of type `TOMBSTONE` contains objectID, so tombstone and tombstoned object must have the same containerID. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-24 17:09:06 +03:00
Dmitrii Stepanov	8434f3dbfc	[#1385] metabase: Use Batch for delete-related operations ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-24 17:09:06 +03:00
Anton Nikiforov	34e6a309c6	[#1356] engine: Evacuate object from shards concurrently ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-09-24 12:50:19 +03:00
Dmitrii Stepanov	0b87be804a	[#1381] engine: Fix tests ... Drop not required `Eventually` calls. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-23 11:10:29 +03:00
Aleksey Savchuk	c34b8acedd	[#1312] Drop handling of system attributes with NeoFS prefix ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-20 11:37:18 +00:00
Dmitrii Stepanov	e3764c51df	[#1347] metabase: Fix EC search ... For EC chunks need to return EC parent object ID as EC chunks don't have own attributes but inherit parent's. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 13:23:29 +00:00
Dmitrii Stepanov	b33559754d	[#1367] fstree: Add size hint for Delete ... This allow to not to call `os.Stat` if caller already knows data size. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	f345fe9a58	[#1367] writecache: Move DB related code to upgrade.go ... This is done to drop this file in the future. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	3b236160a6	[#1367] writecache: Drop DB label from metrics ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	25d2ae8aaf	[#1367] writecache: Drop BBolt related config variables ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	e39378b1c3	[#1367] writecache: Add background flushing objects limiter ... To limit memory usage by background flush. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	8a6e3025a0	[#1367] writecache: Flush from FSTree concurrently ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	2dd3a6f7a8	[#1367] fstree: Add IterateInfo method ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	b142b6f48e	[#1367] fstree: Add size to file counter ... FSTree file counter used by writecache. As writecache has now only one storage, so it is required to use real object size to get writecache size more accurate than `count * max_object_size`. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	5f6c7cbdb1	[#1367] writecache: Drop bbolt DB ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 14:22:29 +03:00
Dmitrii Stepanov	92fe5d90f5	[#1359] writecache: Drop unused ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-09 18:45:21 +03:00
Dmitrii Stepanov	4668efc0bf	[#1355] metabase: Upgrade improvements ... Do not fail on same latest version to run compact on upgraded metabase. Use NoSync on compact. Log every batch on bucket delete stage. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-09 11:37:36 +03:00
Dmitrii Stepanov	d3b209c8e1	[#1337] shard: Disable background rebuild ... Since `frostfs-cli control shards rebuild` command was added, there is no need for background rebuild now. For failover tests used used value 1 to rebuild only schema change. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 15:19:55 +03:00
Dmitrii Stepanov	edb1747af7	[#1337] blobovniczatree: Add rebuild by overflow ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 15:19:54 +03:00
Dmitrii Stepanov	a61201a987	[#1337] config: Move rebuild_worker_count to shard section ... This makes it simple to limit performance degradation for every shard because of rebuild. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 13:57:27 +03:00
Dmitrii Stepanov	6b6eabe41c	[#1337] cli: Add control shards rebuild command ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 13:57:27 +03:00
Dmitrii Stepanov	d508da8397	[#1337] blobovniczatree: Add rebuild by fill percent ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 13:57:27 +03:00
Dmitrii Stepanov	007827255e	[#1337] blobovniczatree: Add .rebuild temp files ... This allows to reduce open/close DBs to check incompleted rebuilds. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 13:57:27 +03:00
Anton Nikiforov	108e4e07be	[#1349] node: Evacuate objects without setting mode to MAINTENANCE ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-09-05 16:08:27 +03:00
Ekaterina Lebedeva	a685fcdc96	[#1317] go.mod: Use range over int ... Since Go 1.22 a "for" statement with a "range" clause is able to iterate through integer values from zero to an upper limit. gopatch script: @@ var i, e expression @@ -for i := 0; i <= e - 1; i++ { +for i := range e { ... } @@ var i, e expression @@ -for i := 0; i <= e; i++ { +for i := range e + 1 { ... } @@ var i, e expression @@ -for i := 0; i < e; i++ { +for i := range e { ... } Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-09-03 13:00:54 +03:00
Aleksey Savchuk	2b3fc50681	[#1320] shard: Fix TestGCDropsObjectInhumedFromWritecache flaky test ... The `TestGCDropsObjectInhumedFromWritecache` test was flaky because a running asynchronous rebuild operation prevented GC from deleting the object. A test-only shard option `WithDisabledRebuild` has been added to fix this. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-02 10:26:53 +03:00
Dmitrii Stepanov	882c068410	[#1334] metabase: Store upgrade flag ... This allows to check if metabase upgrade was not completed. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-30 09:06:20 +03:00
Dmitrii Stepanov	6c2146bbc1	[#1334] metabase: Add upgrade from v2 to v3 ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-30 09:06:20 +03:00
Evgenii Stratonikov	03976c6ed5	[#1341] .golangci.yml: Replace exportloopref with copyloopvar ... exportloopref is deprecated. gopatch: ``` @@ var index, value identifier var slice expression @@ for index, value := range slice { ... -value := value ... } @@ var index, value identifier var slice expression @@ for index, value := range slice { ... -index := index ... } @@ var value identifier var channel expression @@ for value := range channel { ... -value := value ... } ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-28 15:44:41 +00:00
Evgenii Stratonikov	806ea37101	[#1328] pilorama: Do not skip items in SortedByFilename ... Benchmark results: ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ new │ │ sec/op │ sec/op vs base │ ForestSortedIteration/bbolt,root-8 207.2µ ± 6% 173.6µ ± 6% -16.23% (p=0.000 n=10) ForestSortedIteration/bbolt,leaf-8 3.910µ ± 5% 3.928µ ± 7% ~ (p=0.529 n=10) geomean 28.46µ 26.11µ -8.27% ``` They are not representative, as the worst case is when we have multiple items of different lengths. However, `FileName` is usually less than 100 in practice, so the asymptotics is the same. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-26 06:11:32 +00:00