5131 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Dmitrii Stepanov	e39378b1c3	[#1367] writecache: Add background flushing objects limiter ... To limit memory usage by background flush. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	8a6e3025a0	[#1367] writecache: Flush from FSTree concurrently ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	2dd3a6f7a8	[#1367] fstree: Add IterateInfo method ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	b142b6f48e	[#1367] fstree: Add size to file counter ... FSTree file counter used by writecache. As writecache has now only one storage, so it is required to use real object size to get writecache size more accurate than `count * max_object_size`. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 15:06:33 +03:00
Dmitrii Stepanov	5f6c7cbdb1	[#1367] writecache: Drop bbolt DB ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-12 14:22:29 +03:00
Aleksey Savchuk	66e17f4b8e	[#1368] cli/container: Use dedicated method to list user attributes ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-12 10:01:27 +03:00
Aleksey Savchuk	99be4c83a7	[#1368] *: Run gofumpt ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-12 10:00:28 +03:00
Vitaliy Potyarkin	ec8da40567	[#1369] Update obsolete URLs ... Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>	2024-09-11 11:31:10 +00:00
Ekaterina Lebedeva	dea6f031f9	[#1331] cli/tree: Add order flag to tree get-subtree ... Added `--ordered` flag to sort output by ascending FileName. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-09-11 11:30:28 +00:00
Aleksey Savchuk	5fac4058e8	[#1364] cmd/common: Add tests for CreateViper and ReloadViper ... Add tests for `CreateViper` and `ReloadViper` to ensure that no extra files, except *.yaml, *.yml, *.json, are loaded from config directory. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-11 08:03:05 +00:00
Alexander Chuprov	2220f6a809	[#1365] Makefile: Fix HUB_IMAGE ... Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>	2024-09-10 16:45:15 +03:00
Airat Arifullin	a812932984	[#1362] ape: Move common APE check logic to separate package ... * Tree and object service have the same log for checking APE. So, this check should be moved to common package. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-09-10 12:40:34 +00:00
Dmitrii Stepanov	92fe5d90f5	[#1359] writecache: Drop unused ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-09 18:45:21 +03:00
Dmitrii Stepanov	4668efc0bf	[#1355] metabase: Upgrade improvements ... Do not fail on same latest version to run compact on upgraded metabase. Use NoSync on compact. Log every batch on bucket delete stage. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-09 11:37:36 +03:00
Dmitrii Stepanov	654d970fad	[#1355] adm: Run metabase upgrade concurrently ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 15:54:40 +03:00
Dmitrii Stepanov	d3b209c8e1	[#1337] shard: Disable background rebuild ... Since `frostfs-cli control shards rebuild` command was added, there is no need for background rebuild now. For failover tests used used value 1 to rebuild only schema change. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 15:19:55 +03:00
Dmitrii Stepanov	edb1747af7	[#1337] blobovniczatree: Add rebuild by overflow ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 15:19:54 +03:00
Dmitrii Stepanov	a61201a987	[#1337] config: Move rebuild_worker_count to shard section ... This makes it simple to limit performance degradation for every shard because of rebuild. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 13:57:27 +03:00
Dmitrii Stepanov	6b6eabe41c	[#1337] cli: Add control shards rebuild command ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 13:57:27 +03:00
Dmitrii Stepanov	d508da8397	[#1337] blobovniczatree: Add rebuild by fill percent ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 13:57:27 +03:00
Dmitrii Stepanov	007827255e	[#1337] blobovniczatree: Add .rebuild temp files ... This allows to reduce open/close DBs to check incompleted rebuilds. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-09-06 13:57:27 +03:00
Ekaterina Lebedeva	f652518c24	[#1357] go: Fix panic caused by using range over slice len ... If slice is altered in `for` loop, we cannot use range over its length: it may cause panic if slice gets shorter. Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-09-06 13:09:58 +03:00
Airat Arifullin	273980cfb9	[#1310] object: Remove irrelevant comments ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-09-05 16:40:32 +03:00
Anton Nikiforov	108e4e07be	[#1349] node: Evacuate objects without setting mode to MAINTENANCE ... Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>	2024-09-05 16:08:27 +03:00
Airat Arifullin	b3deb893ba	[#1310] object: Move target initialization to separate package ... * Split the logic of write target initialization to different packages; * Refactor patch and put services: since both service initialize the target themselves. Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-09-05 13:03:58 +00:00
Aleksey Savchuk	7768a482b5	[#1223] lens/tui: Add TUI app for blobovnicza ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-05 08:03:52 +00:00
Aleksey Savchuk	371d97f61a	[#1223] lens/tui: Add TUI app for write cache ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-05 08:03:52 +00:00
Aleksey Savchuk	e655336390	[#1223] lens/tui: Add app help ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-05 08:03:52 +00:00
Aleksey Savchuk	ed396448ac	[#1223] lens/tui: Add TUI app to explore metabase ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-05 08:03:52 +00:00
Aleksey Savchuk	9cbd32bce8	[#1223] lens/tui: Add writecache schema ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-05 08:03:52 +00:00
Aleksey Savchuk	1ae86f35a8	[#1223] lens/tui: Add metabase schema ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-05 08:03:52 +00:00
Aleksey Savchuk	b9043433a0	[#1223] scripts: Add script to populate metabase ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-05 08:03:52 +00:00
Aleksey Savchuk	a4fb7f085b	[#1348] go.mod: Update api-go and sdk-go ... Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-04 10:47:26 +03:00
Ekaterina Lebedeva	a685fcdc96	[#1317] go.mod: Use range over int ... Since Go 1.22 a "for" statement with a "range" clause is able to iterate through integer values from zero to an upper limit. gopatch script: @@ var i, e expression @@ -for i := 0; i <= e - 1; i++ { +for i := range e { ... } @@ var i, e expression @@ -for i := 0; i <= e; i++ { +for i := range e + 1 { ... } @@ var i, e expression @@ -for i := 0; i < e; i++ { +for i := range e { ... } Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>	2024-09-03 13:00:54 +03:00
Aleksey Savchuk	2b3fc50681	[#1320] shard: Fix TestGCDropsObjectInhumedFromWritecache flaky test ... The `TestGCDropsObjectInhumedFromWritecache` test was flaky because a running asynchronous rebuild operation prevented GC from deleting the object. A test-only shard option `WithDisabledRebuild` has been added to fix this. Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>	2024-09-02 10:26:53 +03:00
Evgenii Stratonikov	98fe24cdb7	[#1343] go.mod: Update api-go ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-30 08:45:24 +00:00
Dmitrii Stepanov	882c068410	[#1334] metabase: Store upgrade flag ... This allows to check if metabase upgrade was not completed. Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-30 09:06:20 +03:00
Dmitrii Stepanov	6c2146bbc1	[#1334] metabase: Add upgrade from v2 to v3 ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-30 09:06:20 +03:00
Evgenii Stratonikov	03976c6ed5	[#1341] .golangci.yml: Replace exportloopref with copyloopvar ... exportloopref is deprecated. gopatch: ``` @@ var index, value identifier var slice expression @@ for index, value := range slice { ... -value := value ... } @@ var index, value identifier var slice expression @@ for index, value := range slice { ... -index := index ... } @@ var value identifier var channel expression @@ for value := range channel { ... -value := value ... } ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-28 15:44:41 +00:00
Evgenii Stratonikov	7e97df4878	[#1341] Makefile: Update golangci-lint ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-28 15:44:41 +00:00
Evgenii Stratonikov	01b6f1733c	[#1341] Makefile: Build linter with -trimpath ... Fix error with go1.23: ``` Error: build linters: unable to load custom analyzer "truecloudlab-linters": ../linters/bin/external_linters.so, plugin.Open("/repo/frostfs/linters/bin/external_linters"): plugin was built with a different version of package cmp Failed executing command with error: build linters: unable to load custom analyzer "truecloudlab-linters": ../linters/bin/external_linters.so, plugin.Open("/repo/frostfs/linters/bin/external_linters"): plugin was built with a different version of package cmp ``` Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-28 15:44:41 +00:00
Dmitrii Stepanov	7abbdca064	[#1340] getSvc: Fix access denied error handling ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-28 14:02:28 +03:00
Airat Arifullin	6488ddee88	[#1338] object: Fix range provider in Patch handler ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-08-27 16:51:12 +03:00
Airat Arifullin	d6b42972a8	[#1338] object: Fix audit patch stream ... Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>	2024-08-27 16:18:29 +03:00
Evgenii Stratonikov	5e9a97fd3e	[#1336] go.mod: Update api-go and sdk-go ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-27 14:48:43 +03:00
Evgenii Stratonikov	fa7f9fbce2	[#1333] go.mod: Update api-go ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-26 15:37:43 +03:00
Evgenii Stratonikov	806ea37101	[#1328] pilorama: Do not skip items in SortedByFilename ... Benchmark results: ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz │ old │ new │ │ sec/op │ sec/op vs base │ ForestSortedIteration/bbolt,root-8 207.2µ ± 6% 173.6µ ± 6% -16.23% (p=0.000 n=10) ForestSortedIteration/bbolt,leaf-8 3.910µ ± 5% 3.928µ ± 7% ~ (p=0.529 n=10) geomean 28.46µ 26.11µ -8.27% ``` They are not representative, as the worst case is when we have multiple items of different lengths. However, `FileName` is usually less than 100 in practice, so the asymptotics is the same. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-26 06:11:32 +00:00
Evgenii Stratonikov	80099d9a2f	[#1328] pilorama: Add tricky test for SortedByFilename ... Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>	2024-08-26 06:11:32 +00:00
Dmitrii Stepanov	a059a7dcf0	[#1329] cli: Skip linking objects in complex object processing ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-23 13:51:20 +03:00
Dmitrii Stepanov	bd24beecf8	[#1329] putSvc: Reset SuccessAfter for non-EC objects in EC container broadcasting ... Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>	2024-08-23 13:51:16 +03:00