3205 commits

Author	SHA1	Message	Date
Dmitrii Stepanov	1f3e09647a	[#1553] ec: Check all parts are saved ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-12-11 13:21:52 +03:00
Evgenii Stratonikov	ce55e1579b	[#1542] go.mod: Update sdk-go ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-12-06 11:20:16 +03:00
Anton Nikiforov	990e984023	[#1427] object: Fix Put for EC object when node unavailable ... There might be situation when context canceled earlier than traverser move to another part of the nodes. To avoid this, need to wait for the result from concurrent put at each traverser iteration. Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-11-12 12:55:31 +03:00
Ekaterina Lebedeva	8423e0f6f9	[#1316] lint: Fix warnings ... Renamed parameters `min/max` to avoid conflicts with predeclared identifiers. Replaced background context with parent context without cancellation in closer functions in frostfs-node. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com> Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-09-06 17:05:40 +03:00
Dmitrii Stepanov	08593f664b	[#1329] putSvc: Reset SuccessAfter for non-EC objects in EC container broadcasting ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-05 16:11:01 +03:00
Airat Arifullin	eec359cfa8	[#1351] apemanager: Fix AddChain handler for audit middleware ... * `GetChainID` from `frostfs-api-go/v2` does not handler nil-pointer response body that leads to the panic after its call. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-09-04 15:57:34 +03:00
Dmitrii Stepanov	43d6fbf73b	[#1339] getSvc: Fix access denied error handling ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-28 14:00:57 +03:00
Evgenii Stratonikov	01e18eda43	[#1328] pilorama: Do not skip items in SortedByFilename ... Benchmark results: ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ new │ │ sec/op │ sec/op vs base │ ForestSortedIteration/bbolt,root-8 207.2µ ± 6% 173.6µ ± 6% -16.23% (p=0.000 n=10) ForestSortedIteration/bbolt,leaf-8 3.910µ ± 5% 3.928µ ± 7% ~ (p=0.529 n=10) geomean 28.46µ 26.11µ -8.27% ``` They are not representative, as the worst case is when we have multiple items of different lengths. However, `FileName` is usually less than 100 in practice, so the asymptotics is the same. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-23 13:41:01 +03:00
Evgenii Stratonikov	3e89e744aa	[#1328] pilorama: Add tricky test for SortedByFilename ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-23 13:23:15 +03:00
Anton Nikiforov	5aef303259	[#1318] meta: Add test TestInhumeECObject ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-08-20 16:34:42 +03:00
Anton Nikiforov	dd570c9344	[#1318] metrics: Fix container_size_bytes for EC ... When node put chunk into EC container, `policer` may remove it as redundant. This chunk marked as removed. When parent object removed and `gc` start iterating over chunk, node count removing chunk twice. Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-08-20 16:31:59 +03:00
Dmitrii Stepanov	149f8f4b08	[#1319] treeSvc: Do not wrap error from APE ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-19 18:39:25 +03:00
Ekaterina Lebedeva	f82b7e1ae3	[#1135] ir: Add healthstatus RECONFIGURING ... Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-08-15 18:24:10 +03:00
Evgenii Stratonikov	4c520be9f1	[#1313] blobovnicza: Prevent concurrent Put/Close ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-15 15:49:07 +03:00
Dmitrii Stepanov	8b4f3d82c9	[#1302] putSvc: Override SuccessAfter for non-regular objects in EC containers ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-09 10:06:18 +00:00
Dmitrii Stepanov	078b5fdd6e	[#1302] writecache: Allow to specify custom page size ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-09 12:39:11 +03:00
Dmitrii Stepanov	b3ab9589a5	[#1302] writecache: Add put->flush->put benchmark ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-09 12:35:05 +03:00
Dmitrii Stepanov	54862250b2	[#1295] getSvc: Assemble complex EC object headers without linking object ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-06 17:51:23 +03:00
Dmitrii Stepanov	89d670c2f0	[#1295] engine: Resolve funlen linter ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-06 17:51:06 +03:00
Dmitrii Stepanov	87c5954f4e	[#1295] engine: Log object address in case of error ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-06 17:50:48 +03:00
Dmitrii Stepanov	28fc41bd98	[#1291] morph: Reconnect to the highest priority endpoint ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-05 13:17:44 +03:00
Dmitrii Stepanov	20f308876c	[#1288] putSvc: Respect TTL for EC put ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-02 17:55:54 +03:00
Dmitrii Stepanov	f53d30fa95	[#1278] containerSvc: Validate FrostFSID subject exitence on Put ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-29 16:05:10 +03:00
Dmitrii Stepanov	1b92817bd3	[#1278] ir: Do not allow to create container without FrostFSID record ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-29 16:04:53 +03:00
Evgenii Stratonikov	32ec421ac7	[#1277] go.mod: Update api-go ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-26 17:27:41 +03:00
Dmitrii Stepanov	308da7cb01	[#1271] getSvc: Fix local EC chunk get ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-24 10:23:35 +03:00
Dmitrii Stepanov	37b83c0856	[#1271] getSvc: Fix head --raw assemble for EC ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-24 10:23:15 +03:00
Dmitrii Stepanov	18182e578e	[#1261] shard: Fix delete objects from FSTree ... Replace nil storageID with empty like by shard.Get. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-22 14:38:09 +03:00
Dmitrii Stepanov	3119f2fbc3	[#1257] metabase: Delete EC gc marks and split info ... EC parent and split gc marks should be deleted after last EC chunk delete. Also should delete split info for EC parent and split parent. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-19 17:14:14 +03:00
Airat Arifullin	eadcea8df0	[#1249] object: Remove all APE pre-checks in handlers ... * Methods `Head`, `Get`, `GetRangeHash` should no longer use APE pre-checks as that leads only to incorrect rule chain processing for requests: 1. Immediate return with `NoRuleFound` may be unexpected as some `Allow` rule is actually defined but can't be matched yet as it gets no object attributes; 2. Immdediate return with `Allow` may be incorrect as some `Deny` rule is actually defined but can't bet matched yet as it gets no object attirbutes; 3. Pre-check breaks compatibility for converted EACL-tables. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-07-18 13:52:43 +00:00
Dmitrii Stepanov	5e5ee545b6	[#1254] policer: Fix svacer warning ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-18 11:30:39 +00:00
Dmitrii Stepanov	27caa8a72f	[#1256] metabase: Put split parent object ID for EC chunks ... It is required to save split parent ID too, not only split ID. Otherwise inhume operation works incorrect: shard with last part may be skipped and parent object will be available. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-17 18:08:07 +03:00
Dmitrii Stepanov	e83d39e33f	[#1253] deleteSvc: Use copy of common parameters ... getSvc may change the values of some fields, so Head will affect Delete or Put. In this case, the change is necessary so that the session token is stored in the tombstone object (EC assemble calls `ForgetTokens`). Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-17 14:24:27 +03:00
Dmitrii Stepanov	fc383ea6ae	[#1253] getSvc: Fix EC objects get ... Now EC objects assembling is performed concurrently. Also fixed issue with an error in case of getting EC object via non-container node. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-17 14:24:27 +03:00
Evgenii Stratonikov	00a88b9936	[#1251] *: Run gofumpt ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-17 11:25:07 +03:00
Evgenii Stratonikov	3940bc17c1	[#1251] pilorama: Allow traversing multiple branches in parallel ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-17 11:25:07 +03:00
Evgenii Stratonikov	e5767c9002	[#1250] *: Reformat proto filets with clang-format ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-16 15:13:08 +00:00
Evgenii Stratonikov	286df198c9	[#1248] placement: Decouple ContainerNodes() cache from the placement builder. ... Also, write tests. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-16 12:21:30 +00:00
Evgenii Stratonikov	3a48b282b6	[#1248] placement: Use epoch to track netmap versions ... Previously we used pointer, this could have worked, because most of the time, the netmap is cached. This didn't work, however, because `lastNm` field was always nil. Rework the mechanism completely: 1. Use epoch to track netmap versions, as it it simpler and is unrelated to the TTL of an underlying cache. 2. Fix a bug where the epoch could change while mutex was unlocked. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-16 12:21:30 +00:00
Evgenii Stratonikov	21431f22c0	[#1248] placement: Use cid.ID as key in the cache ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-16 12:21:30 +00:00
Airat Arifullin	d5dc14c639	[#1243] object: Make APE checker set x-headers to request properties ... * Update go.mod, go.sum; * Add x-headers to request properties; * Add a unit-test. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-07-16 07:28:42 +00:00
Ekaterina Lebedeva	39866a957c	[#1196] morph/client: Fix ArrayFromStackItem() description ... Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-07-15 16:24:58 +03:00
Ekaterina Lebedeva	d02a7c2d38	[#1196] morph/client: Remove duplicate utility functions ... * We used several utility functions to parse frostfsid client subject and extended subject. However, following the changes in TrueCloudLab/frostfs-contract#97, these utility functions have become public. So there is no more need to have them here. * There was a mismatch of slice parameter required length between frostfs-node's and frostfs-contract's utility functions, `checkStackItem()` solves this problem. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-07-15 16:24:48 +03:00
Anton Nikiforov	3bf6e6dde6	[#1238] engine: Add non-blocking send in the shard's notification channel ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-07-10 11:37:11 +03:00
Evgenii Stratonikov	b027a7f91e	[#1234] pilorama: Fix GetByPath() on duplicate directories ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-10 06:11:38 +00:00
Evgenii Stratonikov	6ace2f597e	[#1234] pilorama: Add test for duplicate directory behaviour ... When AddByPath() is called concurrently on 2 different nodes, internal path components may be created twice. This violates some of our assumptions in GetByPath() and, indirectly, in S3 handling of GetSubTree() results. Add a test for the correct behaviour, fixes will follow. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-07-10 06:11:38 +00:00
Dmitrii Stepanov	84ecd61dfd	[#1233] putSvc: Try to put EC chunk to any node ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-07-09 07:54:29 +00:00
Airat Arifullin	d90aab5454	[#1229] util: Fix session token expiration check ... * Make session token expired at `current_epoch + 1` but not at `current_epoch` when it's still valid. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-07-08 08:15:56 +00:00
Airat Arifullin	0c2b6f3dac	[#1216] ape: Make services use bearer chains fed router ... * Refactor object and tree service - they should instantiate chain router cheking the bearer token. If there are no bearer token rules, then defaul chain router is used. * Fix unit-tests. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-07-05 18:26:48 +00:00
Airat Arifullin	47bcd346d3	[#1216] ape: Introduce BearerChainFeedRouter ... * Unlike default chain router, `BearerChainFedRouter` performs checks for overrides defined in the bearer token; * Add unit-test for the introduced router. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-07-05 18:26:48 +00:00