1226 commits

Author	SHA1	Message	Date
Aleksey Savchuk	281d65435e	[#1450] engine: Group object by shard before Inhume ... ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine cpu: 12th Gen Intel(R) Core(TM) i5-1235U │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ InhumeMultipart/objects=1-12 11.42m ± 1% 10.71m ± 0% -6.27% (p=0.000 n=10) InhumeMultipart/objects=10-12 113.5m ± 0% 100.9m ± 3% -11.08% (p=0.000 n=10) InhumeMultipart/objects=100-12 1135.4m ± 1% 681.3m ± 2% -40.00% (p=0.000 n=10) InhumeMultipart/objects=1000-12 11.358 ± 0% 1.089 ± 1% -90.41% (p=0.000 n=10) InhumeMultipart/objects=10000-12 113.251 ± 0% 1.645 ± 1% -98.55% (p=0.000 n=10) geomean 1.136 265.5m -76.63% ``` Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-04 10:09:00 +03:00
Aleksey Savchuk	b348b20289	[#1450] engine: Add benchmark for Inhume operation ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-04 10:08:34 +03:00
Aleksey Savchuk	748edd1999	[#1450] engine: Return shard-level error if object is expired on inhume ... Since we have errors defined on the shard-level, it looks strage that we check an error againt the shard-level error `ErrLockObjectRemoval`, but then return the metabase-level error. Let's return the same shard-level error instead. Since we have errors defined on the shard-level Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-04 10:06:57 +03:00
Aleksey Savchuk	432042c534	[#1527] engine: Add tests for handling expired objects on inhume and lock ... Currently, it's allowed to inhume or lock an expired object. Consider the following scenario: 1) An user inhumes or locks an object 2) The object expires 3) GC hasn't yet deleted the object 4) The node loses the associated tombstone or lock 5) Another node replicates tombstone or lock to the first node In this case, the second node succeeds, which is the desired behavior. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-03 12:29:45 +03:00
Aleksey Savchuk	9cabca9dfe	[#1527] engine/test: Move default metabase options to separate function ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-02 16:13:37 +03:00
Aleksey Savchuk	60feed3b5f	[#1527] engine/test: Allow to specify current epoch in epochState ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-12-02 15:37:25 +03:00
Aleksey Savchuk	e0ac3a583f	[#1523] metabase: Remove (*DB).IterateCoveredByTombstones ... Remove this method because it isn't used anywhere since 7799f8e4c. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-29 10:49:24 +00:00
Evgenii Stratonikov	01acec708f	[#1525] pilorama: Use AppendUint* helpers from stdlib ... gopatch: ``` @@ var slice, e expression @@ +import "encoding/binary" -append(slice, byte(e), byte(e >> 8)) +binary.LittleEndian.AppendUint16(slice, e) ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-28 09:40:20 +03:00
Evgenii Stratonikov	d77a218f7c	[#1493] metabase: Merge Inhume() and DropGraves() for tombstones ... DropGraves() is only used to drop gravemarks after a tombstone removal. Thus, it makes sense to do Inhume() and DropGraves() in one transaction. It has less overhead and no unexpected problems in case of sudden power failure. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-14 06:47:04 +00:00
Evgenii Stratonikov	44df67492f	[#1493] metabase: Split inhumeTx() into 2 functions ... No functional changes. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-14 06:47:04 +00:00
Evgenii Stratonikov	1e6f132b4e	[#1493] metabase: Pass InhumePrm by value ... Unify with the other code, no functional changes. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-14 06:47:04 +00:00
Evgenii Stratonikov	6dc0dc6691	[#1493] shard: Take mode mutex in HandleExpiredTombstones() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-14 06:47:04 +00:00
Aleksey Savchuk	7fc6101bec	[#1491] engine/test: Rework engine test utils ... - Remove `testNewShard` and `setInitializedShards` because they violated the default engine workflow. The correct workflow is: first use `New()`, followed by `Open()`, and then `Init()`. As a result, adding new logic to `(*StorageEngine).Init` caused several tests to fail with a panic when attempting to access uninitialized resources. Now, all engines created with the test utils must be initialized manually. The new helper method `prepare` can be used for that purpose. - Additionally, `setInitializedShards` hardcoded the shard worker pool size, which prevented it from being configured in tests and benchmarks. This has been fixed as well. - Ensure engine initialization is done wherever it was missing. - Refactor `setShardsNumOpts`, `setShardsNumAdditionalOpts`, and `setShardsNum`. Make them all depend on `setShardsNumOpts`. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-13 14:42:53 +03:00
Aleksey Savchuk	7ef36749d0	[#1491] engine/test: Move BenchmarkExists to exists_test.go ... Move `BenchmarkExists` from `engine_test.go` to `exists_test.go` for better organization and clarity. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-13 14:09:29 +03:00
Aleksey Savchuk	c6066d6ee4	[#1491] engine/test: Use more suitable testing utils here and there ... Use `setShardsNum` instead of `setInitializedShards` wherever possible. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-13 14:09:29 +03:00
Dmitrii Stepanov	7429553266	[#1437] node: Fix contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:10 +03:00
Dmitrii Stepanov	16598553d9	[#1437] shard: Fix contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:09 +03:00
Dmitrii Stepanov	62b5181618	[#1437] blobovnicza: Fix contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:08 +03:00
Dmitrii Stepanov	6db46257c0	[#1437] node: Use ctx for logging ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:07 +03:00
Evgenii Stratonikov	fd004add00	[#1492] metabase: Fix import formatting ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-13 07:30:25 +00:00
Evgenii Stratonikov	8ed7a676d5	[#1492] metabase: Ensure Unmarshal() is called on a cloned slice ... The slice returned from bucket.Get() is only valid during the tx lifetime. Cloning it is not necessary everywhere, but better safe than sorry. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-13 07:30:25 +00:00
Evgenii Stratonikov	b451de94c8	[#1492] metabase: Fix typo in objData ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-13 07:30:25 +00:00
Dmitrii Stepanov	9bd05e94c8	[#1449] tree: Add ApplyBatch method ... Concurrent Apply can lead to child node applies before parent, so undo/redo operations will perform. This leads to performance degradation in case of tree with many sublevels. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-12 12:11:07 +00:00
Dmitrii Stepanov	8a57c78f5f	[#1484] engine: Fix engine metrics ... 1. Add forgotten metrics for client requests 2. Include execIfNotBlocked into metrics Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-11 12:59:20 +03:00
Airat Arifullin	9b13a18aac	[#1479] go.mod: Bump frostfs-sdk-go version ... * Update version within go.mod; * Fix deprecated frostfs-api-go/v2 package and use frostfs-sdk-go/api instead. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-11-08 10:43:19 +03:00
Evgenii Stratonikov	15102e6dfd	[#1471] Replace sort.Slice in some places ... `slices.SortFunc` doesn't use reflection and is a bit faster. I have done some micro-benchmarks for `[]NodeInfo`: ``` $ benchstat -col "/func" out goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ sort.Slice │ slices.SortFunc │ │ sec/op │ sec/op vs base │ Sort-8 2.130µ ± 2% 1.253µ ± 2% -41.20% (p=0.000 n=10) ``` Haven't included them, though, as they I don't see them being used a lot. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-06 08:07:32 +00:00
Evgenii Stratonikov	c2effcc61c	[#1465] Makefile: Update golangci-lint, fix warnings ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-31 06:39:59 +00:00
Evgenii Stratonikov	7ac0852364	[#1459] .golangci.yml: Add intrange linter, fix issues ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-30 15:18:22 +00:00
Evgenii Stratonikov	b0c5def2d9	[#1433] shard/test: Use WithDisabledGC() option where possible ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-17 14:38:07 +03:00
Alexander Chuprov	41038b2ec0	[#1431] node: Fix 'empty slice declaration using a literal' ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-10-15 08:53:59 +00:00
Alexander Chuprov	d83879d4b8	[#1431] node: Fix comment format ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-10-15 08:53:59 +00:00
Alexander Chuprov	f6582081a4	[#1431] obj_storage/metabase: Delete unused variable ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-10-15 08:53:59 +00:00
Alexander Chuprov	00b1cecfb7	[#1431] obj_storage/shard: Fix visibility of 'newMetricStore' ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-10-15 08:53:59 +00:00
Alexander Chuprov	d53732f663	[#1431] engine: Delete always false condition ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-10-15 08:53:59 +00:00
Alexander Chuprov	3012286452	[#1431] metabase: Fix unreachable code ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-10-15 08:53:59 +00:00
Anton Nikiforov	936ebbb8e5	[#1423] metabase: Hide BucketName form upper levels ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-10-08 18:41:16 +03:00
Dmitrii Stepanov	c065d55ca3	[#1412] metabase: Drop logging inside transaction ... This could lead to hang the db. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:52 +03:00
Dmitrii Stepanov	fe9f664b57	[#1412] metabase: Drop empty user attribute buckets on upgrade ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:52 +03:00
Dmitrii Stepanov	87f4b934d1	[#1412] metabase: Run bucket drop steps on upgrade concurrently ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:52 +03:00
Dmitrii Stepanov	8093e145b3	[#1412] adm: Resolve container type by metabase upgrade ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:51 +03:00
Dmitrii Stepanov	3da168f8cf	[#1412] shard: Resolve container is indexed on metabase resync ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:51 +03:00
Dmitrii Stepanov	4572fa4874	[#1412] searchSvc: Check container is indexed ... For non S3 containers it is expected to use attributes index for some attributes. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:50 +03:00
Dmitrii Stepanov	1efa64ee72	[#1412] metabase: Add search by indexed attributes ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:50 +03:00
Dmitrii Stepanov	be744ae3e6	[#1412] metabase: Index attributes for indexed containers ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:49 +03:00
Dmitrii Stepanov	1b520f7973	[#1412] engine: Add IsIndexedContainer flag ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:49 +03:00
Dmitrii Stepanov	899cd55c27	[#1412] engine: PutPrm refactoring ... Use fields instead of methods. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-10-08 11:41:48 +03:00
Evgenii Stratonikov	fc032838c0	[#1215] blobstor/test: Cover iteration behaviour ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-07 12:44:27 +00:00
Anton Nikiforov	2f710d8f94	[#1414] metabase: Check parameter for CountAliveObjectsInBucket ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-10-04 17:11:15 +03:00
Evgenii Stratonikov	4dc9a1b300	[#1413] engine: Remove error counting methods from Shard ... All error counting and hangling logic is present on the engine level. Currently, we pass engine metrics with shard ID metric to shard, then export 3 methods to manipulate these metrics. In this commits all methods are removed and error counter is tracked on the engine level exlusively. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-04 15:10:17 +03:00
Evgenii Stratonikov	963faa615a	[#1413] engine: Cleanup shard error reporting ... - `reportShardErrorBackground()` no longer differs from `reportShardError()`, reflect this in its name; - reuse common pieces of code to make it simpler. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-10-04 15:10:17 +03:00