TrueCloudLab
/
frostfs-node
18
1
Fork 20
Code Issues 88 Pull Requests 7 Actions Releases Activity

Speed up metabase refill #1027

Merged
fyrchik merged 4 commits from dstepanov-yadro/frostfs-node:feat/resync_speedup into master 2024-04-10 12:08:59 +00:00
Conversation 2 Commits 4 Files Changed 32
dstepanov-yadro commented 2024-03-05 14:00:51 +00:00
Collaborator

Relates #1024
Relates #1029

According to the benchmark results, performance did not increase after the number of concurrently processed objects was 250. I decided to set the default value to 500 so that metabase refill would occur faster on more productive systems. The difference between 250 and 500 goroutines does not look so significant.

Another approach considered is to first read objects batch, and then simultaneously send them for writing to the metabase. But this did not lead to a noticeable increase in productivity.

Also refill metabase metrics were added.

There are benchmark results:

master

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	       1	1943294224 ns/op	113832760 B/op	   56578 allocs/op // 1.9s
BenchmarkRefillMetabase/1000_objects-8        	       1	19000179327 ns/op	1165623184 B/op	  906343 allocs/op // 19.0s
BenchmarkRefillMetabase/2000_objects-8        	       1	38300923157 ns/op	2355198696 B/op	 1920256 allocs/op // 38.3s
BenchmarkRefillMetabase/5000_objects-8        	       1	116061147664 ns/op	5951935552 B/op	 5052396 allocs/op // 116.0s

RefillMetabaseWorkersCountDefault = 10

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.3157 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	3013462568 ns/op	1133285232 B/op	  656850 allocs/op // 3.0s
BenchmarkRefillMetabase/2000_objects-8        	       1	6544228606 ns/op	2248786344 B/op	 1327172 allocs/op // 6.5s
BenchmarkRefillMetabase/5000_objects-8        	       1	15821508113 ns/op	5651548824 B/op	 3450217 allocs/op // 15.8s

RefillMetabaseWorkersCountDefault = 100

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2434 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	2372491712 ns/op	1131595096 B/op	  675049 allocs/op // 2.4s
BenchmarkRefillMetabase/2000_objects-8        	       1	4768436846 ns/op	2240549856 B/op	 1322288 allocs/op // 4.7s
BenchmarkRefillMetabase/5000_objects-8        	       1	11986498868 ns/op	5628292960 B/op	 3383400 allocs/op // 12.0s

RefillMetabaseWorkersCountDefault = 250

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2480 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	2326638880 ns/op	1138337520 B/op	  741418 allocs/op // 2.3s
BenchmarkRefillMetabase/2000_objects-8        	       1	4668489311 ns/op	2245232944 B/op	 1399796 allocs/op // 4.7s
BenchmarkRefillMetabase/5000_objects-8        	       1	11616198474 ns/op	5628608720 B/op	 3457276 allocs/op // 11.6s

RefillMetabaseWorkersCountDefault = 500

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2305 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	2306998473 ns/op	1140297008 B/op	  843014 allocs/op // 2.3s
BenchmarkRefillMetabase/2000_objects-8        	       1	4664998201 ns/op	2255111088 B/op	 1519356 allocs/op // 4.7s
BenchmarkRefillMetabase/5000_objects-8        	       1	11438317463 ns/op	5641908880 B/op	 3599110 allocs/op // 11.4s

RefillMetabaseWorkersCountDefault = 1000

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2355 ns/op	       0 B/op	       0 allocs/op // 0.0s
BenchmarkRefillMetabase/1000_objects-8        	       1	2241713913 ns/op	1137704440 B/op	  840555 allocs/op // 2.2s
BenchmarkRefillMetabase/2000_objects-8        	       1	4681489790 ns/op	2266312320 B/op	 1750873 allocs/op // 4.7s
BenchmarkRefillMetabase/5000_objects-8        	       1	11469093646 ns/op	5658172016 B/op	 3847801 allocs/op // 11.5s

RefillMetabaseWorkersCountDefault = 2000

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2429 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	2269358666 ns/op	1140359296 B/op	  850544 allocs/op // 2.2s
BenchmarkRefillMetabase/2000_objects-8        	       1	4497506710 ns/op	2264781384 B/op	 1772200 allocs/op // 4.5s
BenchmarkRefillMetabase/5000_objects-8        	       1	11692943181 ns/op	5665310944 B/op	 3870899 allocs/op // 11.7s

Metrics dashboards:
image
image

Relates #1024 Relates #1029 According to the benchmark results, performance did not increase after the number of concurrently processed objects was 250. I decided to set the default value to 500 so that metabase refill would occur faster on more productive systems. The difference between 250 and 500 goroutines does not look so significant. Another approach considered is to first read objects batch, and then simultaneously send them for writing to the metabase. But this did not lead to a noticeable increase in productivity. Also refill metabase metrics were added. There are benchmark results: ### master ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1 1943294224 ns/op 113832760 B/op 56578 allocs/op // 1.9s BenchmarkRefillMetabase/1000_objects-8 1 19000179327 ns/op 1165623184 B/op 906343 allocs/op // 19.0s BenchmarkRefillMetabase/2000_objects-8 1 38300923157 ns/op 2355198696 B/op 1920256 allocs/op // 38.3s BenchmarkRefillMetabase/5000_objects-8 1 116061147664 ns/op 5951935552 B/op 5052396 allocs/op // 116.0s ``` ### RefillMetabaseWorkersCountDefault = 10 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.3157 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 3013462568 ns/op 1133285232 B/op 656850 allocs/op // 3.0s BenchmarkRefillMetabase/2000_objects-8 1 6544228606 ns/op 2248786344 B/op 1327172 allocs/op // 6.5s BenchmarkRefillMetabase/5000_objects-8 1 15821508113 ns/op 5651548824 B/op 3450217 allocs/op // 15.8s ``` ### RefillMetabaseWorkersCountDefault = 100 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2434 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 2372491712 ns/op 1131595096 B/op 675049 allocs/op // 2.4s BenchmarkRefillMetabase/2000_objects-8 1 4768436846 ns/op 2240549856 B/op 1322288 allocs/op // 4.7s BenchmarkRefillMetabase/5000_objects-8 1 11986498868 ns/op 5628292960 B/op 3383400 allocs/op // 12.0s ``` ### RefillMetabaseWorkersCountDefault = 250 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2480 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 2326638880 ns/op 1138337520 B/op 741418 allocs/op // 2.3s BenchmarkRefillMetabase/2000_objects-8 1 4668489311 ns/op 2245232944 B/op 1399796 allocs/op // 4.7s BenchmarkRefillMetabase/5000_objects-8 1 11616198474 ns/op 5628608720 B/op 3457276 allocs/op // 11.6s ``` ### RefillMetabaseWorkersCountDefault = 500 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2305 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 2306998473 ns/op 1140297008 B/op 843014 allocs/op // 2.3s BenchmarkRefillMetabase/2000_objects-8 1 4664998201 ns/op 2255111088 B/op 1519356 allocs/op // 4.7s BenchmarkRefillMetabase/5000_objects-8 1 11438317463 ns/op 5641908880 B/op 3599110 allocs/op // 11.4s ``` ### RefillMetabaseWorkersCountDefault = 1000 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2355 ns/op 0 B/op 0 allocs/op // 0.0s BenchmarkRefillMetabase/1000_objects-8 1 2241713913 ns/op 1137704440 B/op 840555 allocs/op // 2.2s BenchmarkRefillMetabase/2000_objects-8 1 4681489790 ns/op 2266312320 B/op 1750873 allocs/op // 4.7s BenchmarkRefillMetabase/5000_objects-8 1 11469093646 ns/op 5658172016 B/op 3847801 allocs/op // 11.5s ``` ### RefillMetabaseWorkersCountDefault = 2000 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2429 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 2269358666 ns/op 1140359296 B/op 850544 allocs/op // 2.2s BenchmarkRefillMetabase/2000_objects-8 1 4497506710 ns/op 2264781384 B/op 1772200 allocs/op // 4.5s BenchmarkRefillMetabase/5000_objects-8 1 11692943181 ns/op 5665310944 B/op 3870899 allocs/op // 11.7s ``` Metrics dashboards: ![image](/attachments/dbe50f16-e643-4866-89d3-7d94dbf28d8a) ![image](/attachments/577f4887-c6ec-4e1e-b265-5171f96a8083)
image.png
211 KiB
image.png
210 KiB
image.png
181 KiB
image.png
dstepanov-yadro reviewed 2024-03-05 14:13:04 +00:00
pkg/local_object_storage/blobovnicza/iterate.go Outdated
@ -160,2 +160,3 @@
if !prm.withoutData {
elem.data = v
data := make([]byte, 0, len(v))
elem.data = append(data, v...)
dstepanov-yadro commented 2024-03-05 14:13:04 +00:00
Poster
Collaborator

v is only valid for tx lifetime. So to use v in separate goroutine it is required to copy it.

`v` is only valid for `tx` lifetime. So to use `v` in separate goroutine it is required to copy it.
fyrchik commented 2024-03-05 14:58:24 +00:00
Owner

bytes.Clone()?

`bytes.Clone()`?
dstepanov-yadro commented 2024-03-05 15:24:29 +00:00
Poster
Collaborator

bytes.Clone() doesn't preallocate enough capacity.

`bytes.Clone()` doesn't preallocate enough capacity.
fyrchik commented 2024-03-05 18:39:37 +00:00
Owner

It does, it may overallocate though, but it is the solution from the stdlib.
Don't mind to leave it as is.

It does, it may overallocate though, but it is the solution from the stdlib. Don't mind to leave it as is.
dstepanov-yadro commented 2024-03-06 07:09:39 +00:00
Poster
Collaborator

fixed

fixed
fyrchik marked this conversation as resolved
dstepanov-yadro requested review from storage-core-committers 2024-03-05 14:21:44 +00:00
dstepanov-yadro requested review from storage-core-developers 2024-03-05 14:21:45 +00:00
acid-ant approved these changes 2024-03-05 14:52:48 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from 8727d55371 to 7e1b5b8c37 2024-03-06 07:09:27 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from 7e1b5b8c37 to 4c479b861d 2024-03-06 07:10:52 +00:00 Compare
acid-ant approved these changes 2024-03-06 07:27:39 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from d92a98923a to 0d91716e81 2024-03-11 13:41:46 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from 0d91716e81 to 78a96c8182 2024-03-12 07:43:34 +00:00 Compare
acid-ant approved these changes 2024-03-12 07:54:03 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from 387ec06718 to a122ef3cd9 2024-03-12 14:38:13 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from a122ef3cd9 to d4e315aba0 2024-03-12 14:53:14 +00:00 Compare
acid-ant reviewed 2024-03-12 15:12:22 +00:00
pkg/local_object_storage/shard/control.go Outdated
@ -187,2 +202,3 @@
obj := objectSDK.New()
withDiskUsage := true
duCtx, cancel := context.WithTimeout(ctx, 10*time.Minute) // if compute disk usage takes too long, then just skip this step
acid-ant commented 2024-03-12 15:12:22 +00:00
Collaborator

Why not to move timeout in config?

Why not to move timeout in config?
dstepanov-yadro commented 2024-03-12 15:18:57 +00:00
Poster
Collaborator

can't imagine the situation when we will change this.

can't imagine the situation when we will change this.
acid-ant approved these changes 2024-03-12 15:19:16 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from d4e315aba0 to dfc2fe2ebf 2024-03-13 07:09:19 +00:00 Compare
aarifullin approved these changes 2024-03-13 10:36:19 +00:00
aarifullin left a comment
Collaborator

LGTM

LGTM
fyrchik commented 2024-04-05 13:37:11 +00:00
Owner

Doesn't this close #1024?

Doesn't this close #1024?
dstepanov-yadro commented 2024-04-08 08:07:00 +00:00
Poster
Collaborator

Doesn't this close #1024?

It does. But I prefer to close tasks manually with comments.

> Doesn't this close #1024? It does. But I prefer to close tasks manually with comments.
dstepanov-yadro force-pushed feat/resync_speedup from dfc2fe2ebf to 182ec28f1c 2024-04-08 08:08:37 +00:00 Compare
fyrchik requested changes 2024-04-08 08:46:46 +00:00
pkg/local_object_storage/blobstor/info.go Outdated
@ -18,0 +49,4 @@
for i := range b.storage {
path := b.storage[i].Storage.Path()
eg.Go(func() error {
return filepath.WalkDir(path, func(path string, d fs.DirEntry, _ error) error {
fyrchik commented 2024-04-08 08:44:43 +00:00
Owner

This won't work for blobovniczas:

  1. They don't reclaim memory and the whole 40mb db could be empty.
  2. B-tree has some overhead for meta.

This becomes worse the more small objects we have.

Have you considered using naive but correct implementation? (traverse blobovniczas, maybe even count objects, not their size).

This won't work for blobovniczas: 1. They don't reclaim memory and the whole 40mb db could be empty. 2. B-tree has some overhead for meta. This becomes worse the more small objects we have. Have you considered using naive but correct implementation? (traverse blobovniczas, maybe even count objects, not their size).
dstepanov-yadro commented 2024-04-08 09:01:49 +00:00
Poster
Collaborator

That's why the method is called DiskUsage, not DataSize.
Every blobovnicza stores data size, but to get this value it is required to open/close db. As it is used only for metrics, I think it is enough to calculate only disk size.

That's why the method is called `DiskUsage`, not `DataSize`. Every blobovnicza stores data size, but to get this value it is required to open/close db. As it is used only for metrics, I think it is enough to calculate only disk size.
fyrchik commented 2024-04-09 15:32:39 +00:00
Owner

But then metrics won't reflect what we deem them reflect (progress of operation).

But then metrics won't reflect what we deem them reflect (progress of operation).
dstepanov-yadro commented 2024-04-10 11:27:10 +00:00
Poster
Collaborator

Agree, fixed.
Now object count used for metrics, data size is hard to compute if compression is enabled.

Agree, fixed. Now object count used for metrics, data size is hard to compute if compression is enabled.
pkg/local_object_storage/shard/control.go Outdated
@ -187,2 +202,3 @@
obj := objectSDK.New()
withDiskUsage := true
duCtx, cancel := context.WithTimeout(ctx, 10*time.Minute) // if compute disk usage takes too long, then just skip this step
fyrchik commented 2024-04-08 08:35:56 +00:00
Owner

It looks strange to me: we can wait for 10 minutes and still do not receive any metrics.
Ironically situations when disk usage calculation takes too long are also likely the ones where having metrics provides a huge QOL improvement. metabase resync duration has the order of tens of hours for big disks, 10 minute is nothing in comparison.

So why don't we allow ourselves hang here?

It looks strange to me: we can wait for 10 minutes and still do not receive any metrics. Ironically situations when disk usage calculation takes too long are also likely the ones where having metrics provides a huge QOL improvement. metabase resync duration has the order of _tens of hours_ for big disks, 10 minute is nothing in comparison. So why don't we allow ourselves hang here?
dstepanov-yadro commented 2024-04-08 09:04:59 +00:00
Poster
Collaborator

Fixed.

Fixed.
fyrchik marked this conversation as resolved
dstepanov-yadro force-pushed feat/resync_speedup from 182ec28f1c to 0306fdc818 2024-04-08 09:04:52 +00:00 Compare
acid-ant approved these changes 2024-04-09 06:58:49 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from 0306fdc818 to 23fdee2e97 2024-04-09 07:19:18 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from 23fdee2e97 to b66ea2c713 2024-04-09 16:48:10 +00:00 Compare
dstepanov-yadro changed title from Speed up metabase refill to WIP: Speed up metabase refill 2024-04-09 16:48:30 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from b66ea2c713 to 6e038936cf 2024-04-10 07:21:47 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from 6e038936cf to 1b17258c04 2024-04-10 11:26:19 +00:00 Compare
dstepanov-yadro changed title from WIP: Speed up metabase refill to Speed up metabase refill 2024-04-10 11:30:16 +00:00
fyrchik approved these changes 2024-04-10 12:08:44 +00:00
fyrchik merged commit 1b17258c04 into master 2024-04-10 12:08:59 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
aarifullin
acid-ant
fyrchik
Labels
Clear labels   
P0

Highest   
P1

High   
P2

Medium   
P3

Low   
badger

Badger DB related issues   
frostfs-adm
   
frostfs-cli
   
frostfs-ir
   
frostfs-lens
   
frostfs-node
   
good first issue

Good for newcomers   
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#1027
There is no content yet.
Delete Branch "dstepanov-yadro/frostfs-node:feat/resync_speedup"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?