5129 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Aleksey Savchuk	59fc75de0b	[#1450] engine: Make Inhume operation handle objects in parallel ... Add a worker pool for `Inhume` operation and use it to handle objects in parallel. Since metabase `Inhume` uses `bbolt.Batch`, handling many objects one by one may be inefficient. ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine cpu: 12th Gen Intel(R) Core(TM) i5-1235U │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ InhumeMultipart/objects=1-12 11.45m ± 0% 11.43m ± 0% ~ (p=0.315 n=10) InhumeMultipart/objects=10-12 113.94m ± 1% 11.60m ± 0% -89.82% (p=0.000 n=10) InhumeMultipart/objects=100-12 1144.08m ± 1% 30.17m ± 12% -97.36% (p=0.000 n=10) InhumeMultipart/objects=1000-12 11367.5m ± 0% 265.7m ± 2% -97.66% (p=0.000 n=10) InhumeMultipart/objects=10000-12 113.505 ± 0% 2.436 ± 1% -97.85% (p=0.000 n=10) geomean 1.140 76.32m -93.31% ``` Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-15 00:50:53 +03:00
Aleksey Savchuk	6eb0ae3be9	[#1450] engine: Add benchmark for Inhume operation ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-15 00:15:19 +03:00
Alexander Chuprov	f7cb6b4d87	[#1482] Makefile: Update golangci-lint ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-11-13 15:01:41 +00:00
Aleksey Savchuk	7fc6101bec	[#1491] engine/test: Rework engine test utils ... - Remove `testNewShard` and `setInitializedShards` because they violated the default engine workflow. The correct workflow is: first use `New()`, followed by `Open()`, and then `Init()`. As a result, adding new logic to `(*StorageEngine).Init` caused several tests to fail with a panic when attempting to access uninitialized resources. Now, all engines created with the test utils must be initialized manually. The new helper method `prepare` can be used for that purpose. - Additionally, `setInitializedShards` hardcoded the shard worker pool size, which prevented it from being configured in tests and benchmarks. This has been fixed as well. - Ensure engine initialization is done wherever it was missing. - Refactor `setShardsNumOpts`, `setShardsNumAdditionalOpts`, and `setShardsNum`. Make them all depend on `setShardsNumOpts`. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-13 14:42:53 +03:00
Aleksey Savchuk	7ef36749d0	[#1491] engine/test: Move BenchmarkExists to exists_test.go ... Move `BenchmarkExists` from `engine_test.go` to `exists_test.go` for better organization and clarity. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-13 14:09:29 +03:00
Aleksey Savchuk	c6066d6ee4	[#1491] engine/test: Use more suitable testing utils here and there ... Use `setShardsNum` instead of `setInitializedShards` wherever possible. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-11-13 14:09:29 +03:00
Dmitrii Stepanov	612b34d570	[#1437] logger: Add caller skip to log original caller position ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:12 +03:00
Dmitrii Stepanov	7429553266	[#1437] node: Fix contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:10 +03:00
Dmitrii Stepanov	6921a89061	[#1437] ir: Fix contextcheck linters ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:10 +03:00
Dmitrii Stepanov	16598553d9	[#1437] shard: Fix contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:09 +03:00
Dmitrii Stepanov	c139892117	[#1437] ir: Fix contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:09 +03:00
Dmitrii Stepanov	62b5181618	[#1437] blobovnicza: Fix contextcheck linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:08 +03:00
Dmitrii Stepanov	6db46257c0	[#1437] node: Use ctx for logging ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-13 10:36:07 +03:00
Dmitrii Stepanov	c16dae8b4d	[#1437] logger: Use context to log trace id ... Signed-off-by: Dmitrii Stepanov	2024-11-13 10:36:07 +03:00
Evgenii Stratonikov	fd004add00	[#1492] metabase: Fix import formatting ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-13 07:30:25 +00:00
Evgenii Stratonikov	8ed7a676d5	[#1492] metabase: Ensure Unmarshal() is called on a cloned slice ... The slice returned from bucket.Get() is only valid during the tx lifetime. Cloning it is not necessary everywhere, but better safe than sorry. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-13 07:30:25 +00:00
Evgenii Stratonikov	b451de94c8	[#1492] metabase: Fix typo in objData ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-13 07:30:25 +00:00
Dmitrii Stepanov	f1556e3c42	[#1488] Makefile: Drop all containers created on env-up ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-12 17:24:44 +03:00
Dmitrii Stepanov	e122ff6013	[#1488] dev: Add Jaeger image and enable tracing on debug ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-12 17:24:44 +03:00
Dmitrii Stepanov	e2658c7519	[#1488] tracing: KV attributes for spans from config ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-12 17:24:43 +03:00
Dmitrii Stepanov	c00f4bab18	[#1488] go.mod: Bump observability version ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-12 17:24:43 +03:00
Dmitrii Stepanov	46fef276b4	[#1449] tree: Log tree sync with Info level ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-12 12:11:07 +00:00
Dmitrii Stepanov	9bd05e94c8	[#1449] tree: Add ApplyBatch method ... Concurrent Apply can lead to child node applies before parent, so undo/redo operations will perform. This leads to performance degradation in case of tree with many sublevels. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-12 12:11:07 +00:00
Evgenii Stratonikov	16830033f8	[#1483] cli: Remove --basic-acl flag ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-12 12:10:51 +00:00
Evgenii Stratonikov	1cf51a8079	[#1483] cli/docs: Remove set-eacl mention ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-12 12:10:51 +00:00
Evgenii Stratonikov	3324c26fd8	[#1483] morph: Remove container.GetEACL() ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-12 12:10:51 +00:00
Evgenii Stratonikov	a692298533	[#1483] node: Remove eACL cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-12 12:10:51 +00:00
Anton Nikiforov	be2753de00	[#1490] docs: Update description for object.get.priority ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-11-12 13:35:13 +03:00
Airat Arifullin	b543569c3f	[#1486] node: Introduce dual service support ... * Register GRPC services for both neo.fs.v2 and frost.fs namespaces * Use this temporary solution until all nodes are updated Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-11-12 08:43:47 +00:00
Alexander Chuprov	80f8a8fd3a	[#1396] cli/playground: Refactor ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-11-12 11:09:27 +03:00
Alexander Chuprov	2f3bc6eb84	[#1396] cli/playground: Improve terminal control key handling ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-11-12 11:07:48 +03:00
Dmitrii Stepanov	8a57c78f5f	[#1484] engine: Fix engine metrics ... 1. Add forgotten metrics for client requests 2. Include execIfNotBlocked into metrics Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-11 12:59:20 +03:00
Vitaliy Potyarkin	ad01fb958a	[#1474] Stop using obsolete .github directory ... This commit is a part of multi-repo cleanup effort: TrueCloudLab/frostfs-infra#136 Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>	2024-11-11 06:44:13 +00:00
Airat Arifullin	d336f2d487	[#1393] adm: Make NewLocalActor receive accout name ... * Some RPC-clients for contracts require different wallet account types. Since, `Policy` contract gets `consensus` accounts while `NNS` gets `committee` accounts. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-11-08 13:57:51 +00:00
Evgenii Stratonikov	c82c753e9f	[#1480] objsvc: Remove useless stream wrappers ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-08 12:01:14 +00:00
Evgenii Stratonikov	f666898e5d	[#1480] objsvc: Remove EACL checks ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-08 12:01:14 +00:00
Evgenii Stratonikov	b1a31281e4	[#1480] ape: Remove SoftAPECheck flag ... Previous release was EACL-compatible. Starting from now all EACL should've been migrated to APE chains. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-08 12:01:14 +00:00
Airat Arifullin	764450d04a	[#1479] tree: Regenerate service protobufs ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-11-08 10:43:44 +03:00
Airat Arifullin	755cae3f19	[#1479] control: Regenerate protobufs for service ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-11-08 10:43:44 +03:00
Airat Arifullin	9b13a18aac	[#1479] go.mod: Bump frostfs-sdk-go version ... * Update version within go.mod; * Fix deprecated frostfs-api-go/v2 package and use frostfs-sdk-go/api instead. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-11-08 10:43:19 +03:00
Dmitrii Stepanov	ef64930fef	[#1477] ape: Fix EC chunk test ... Initially, this test was a check that only the container node can assemble an EC object. But the implementation of this test was wrong. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-07 16:00:20 +03:00
Evgenii Stratonikov	c8fb154151	[#1475] Remove container estimation code ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-06 15:32:33 +03:00
Dmitrii Stepanov	7edec9193c	[#1451] placement: Return copy of slice from container nodes cache ... Nodes from cache could be changed by traverser, if no objectID specified. So it is required to return copy of cache's slice. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-06 08:18:10 +00:00
Dmitrii Stepanov	3cf6ea745d	[#1451] ec: Check all parts are saved ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-06 08:18:10 +00:00
Dmitrii Stepanov	9a77527f46	[#1451] ape: Drop unused ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-06 08:18:10 +00:00
Dmitrii Stepanov	5b1ba8e23d	[#1451] ape: Perform strict APE checks for EC parts ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-06 08:18:10 +00:00
Dmitrii Stepanov	9902965ff4	[#1451] writer: Sign EC parts with node's private key ... As EC put request may be processed only by container node, so sign requests with current node private to not to perform APE checks. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-06 08:18:10 +00:00
Dmitrii Stepanov	33ad753302	[#1473] policer: Add tracing span ... To filter HEAD requests from policer. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-11-06 08:15:11 +00:00
Evgenii Stratonikov	15102e6dfd	[#1471] Replace sort.Slice in some places ... `slices.SortFunc` doesn't use reflection and is a bit faster. I have done some micro-benchmarks for `[]NodeInfo`: ``` $ benchstat -col "/func" out goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ sort.Slice │ slices.SortFunc │ │ sec/op │ sec/op vs base │ Sort-8 2.130µ ± 2% 1.253µ ± 2% -41.20% (p=0.000 n=10) ``` Haven't included them, though, as they I don't see them being used a lot. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-06 08:07:32 +00:00
Evgenii Stratonikov	17ec84151b	[#532] cli: Respect XDG base directory spec ... XDG base directory specification defines where various files should be looked by an application. Hopefully, this makes `frostfs-cli` more predictable and pleasant to work with. Luckily for us, golang already has everything we need in the stdlib. This commit also gets rid of `github.com/mitchellh/go-homedir` dependency. Close #532 Refs #1455 Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-11-05 16:27:18 +03:00