TrueCloudLab/frostfs-node
21
1
Fork 27
Code Issues 108 Pull requests 3 Projects Releases 1 Activity Actions

Speed up metabase refill #1027

Merged
fyrchik merged 4 commits from dstepanov-yadro/frostfs-node:feat/resync_speedup into master 2024-09-04 19:51:07 +00:00
Conversation 2 Commits 4 Files changed 32 +506 -60
dstepanov-yadro commented 2024-03-05 14:00:51 +00:00
Member
Copy link

Relates #1024
Relates #1029

According to the benchmark results, performance did not increase after the number of concurrently processed objects was 250. I decided to set the default value to 500 so that metabase refill would occur faster on more productive systems. The difference between 250 and 500 goroutines does not look so significant.

Another approach considered is to first read objects batch, and then simultaneously send them for writing to the metabase. But this did not lead to a noticeable increase in productivity.

Also refill metabase metrics were added.

There are benchmark results:

master

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	       1	1943294224 ns/op	113832760 B/op	   56578 allocs/op // 1.9s
BenchmarkRefillMetabase/1000_objects-8        	       1	19000179327 ns/op	1165623184 B/op	  906343 allocs/op // 19.0s
BenchmarkRefillMetabase/2000_objects-8        	       1	38300923157 ns/op	2355198696 B/op	 1920256 allocs/op // 38.3s
BenchmarkRefillMetabase/5000_objects-8        	       1	116061147664 ns/op	5951935552 B/op	 5052396 allocs/op // 116.0s

RefillMetabaseWorkersCountDefault = 10

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.3157 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	3013462568 ns/op	1133285232 B/op	  656850 allocs/op // 3.0s
BenchmarkRefillMetabase/2000_objects-8        	       1	6544228606 ns/op	2248786344 B/op	 1327172 allocs/op // 6.5s
BenchmarkRefillMetabase/5000_objects-8        	       1	15821508113 ns/op	5651548824 B/op	 3450217 allocs/op // 15.8s

RefillMetabaseWorkersCountDefault = 100

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2434 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	2372491712 ns/op	1131595096 B/op	  675049 allocs/op // 2.4s
BenchmarkRefillMetabase/2000_objects-8        	       1	4768436846 ns/op	2240549856 B/op	 1322288 allocs/op // 4.7s
BenchmarkRefillMetabase/5000_objects-8        	       1	11986498868 ns/op	5628292960 B/op	 3383400 allocs/op // 12.0s

RefillMetabaseWorkersCountDefault = 250

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2480 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	2326638880 ns/op	1138337520 B/op	  741418 allocs/op // 2.3s
BenchmarkRefillMetabase/2000_objects-8        	       1	4668489311 ns/op	2245232944 B/op	 1399796 allocs/op // 4.7s
BenchmarkRefillMetabase/5000_objects-8        	       1	11616198474 ns/op	5628608720 B/op	 3457276 allocs/op // 11.6s

RefillMetabaseWorkersCountDefault = 500

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2305 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	2306998473 ns/op	1140297008 B/op	  843014 allocs/op // 2.3s
BenchmarkRefillMetabase/2000_objects-8        	       1	4664998201 ns/op	2255111088 B/op	 1519356 allocs/op // 4.7s
BenchmarkRefillMetabase/5000_objects-8        	       1	11438317463 ns/op	5641908880 B/op	 3599110 allocs/op // 11.4s

RefillMetabaseWorkersCountDefault = 1000

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2355 ns/op	       0 B/op	       0 allocs/op // 0.0s
BenchmarkRefillMetabase/1000_objects-8        	       1	2241713913 ns/op	1137704440 B/op	  840555 allocs/op // 2.2s
BenchmarkRefillMetabase/2000_objects-8        	       1	4681489790 ns/op	2266312320 B/op	 1750873 allocs/op // 4.7s
BenchmarkRefillMetabase/5000_objects-8        	       1	11469093646 ns/op	5658172016 B/op	 3847801 allocs/op // 11.5s

RefillMetabaseWorkersCountDefault = 2000

goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkRefillMetabase/100_objects-8         	1000000000	         0.2429 ns/op	       0 B/op	       0 allocs/op
BenchmarkRefillMetabase/1000_objects-8        	       1	2269358666 ns/op	1140359296 B/op	  850544 allocs/op // 2.2s
BenchmarkRefillMetabase/2000_objects-8        	       1	4497506710 ns/op	2264781384 B/op	 1772200 allocs/op // 4.5s
BenchmarkRefillMetabase/5000_objects-8        	       1	11692943181 ns/op	5665310944 B/op	 3870899 allocs/op // 11.7s

Metrics dashboards:
image
image

Relates #1024 Relates #1029 According to the benchmark results, performance did not increase after the number of concurrently processed objects was 250. I decided to set the default value to 500 so that metabase refill would occur faster on more productive systems. The difference between 250 and 500 goroutines does not look so significant. Another approach considered is to first read objects batch, and then simultaneously send them for writing to the metabase. But this did not lead to a noticeable increase in productivity. Also refill metabase metrics were added. There are benchmark results: ### master ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1 1943294224 ns/op 113832760 B/op 56578 allocs/op // 1.9s BenchmarkRefillMetabase/1000_objects-8 1 19000179327 ns/op 1165623184 B/op 906343 allocs/op // 19.0s BenchmarkRefillMetabase/2000_objects-8 1 38300923157 ns/op 2355198696 B/op 1920256 allocs/op // 38.3s BenchmarkRefillMetabase/5000_objects-8 1 116061147664 ns/op 5951935552 B/op 5052396 allocs/op // 116.0s ``` ### RefillMetabaseWorkersCountDefault = 10 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.3157 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 3013462568 ns/op 1133285232 B/op 656850 allocs/op // 3.0s BenchmarkRefillMetabase/2000_objects-8 1 6544228606 ns/op 2248786344 B/op 1327172 allocs/op // 6.5s BenchmarkRefillMetabase/5000_objects-8 1 15821508113 ns/op 5651548824 B/op 3450217 allocs/op // 15.8s ``` ### RefillMetabaseWorkersCountDefault = 100 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2434 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 2372491712 ns/op 1131595096 B/op 675049 allocs/op // 2.4s BenchmarkRefillMetabase/2000_objects-8 1 4768436846 ns/op 2240549856 B/op 1322288 allocs/op // 4.7s BenchmarkRefillMetabase/5000_objects-8 1 11986498868 ns/op 5628292960 B/op 3383400 allocs/op // 12.0s ``` ### RefillMetabaseWorkersCountDefault = 250 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2480 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 2326638880 ns/op 1138337520 B/op 741418 allocs/op // 2.3s BenchmarkRefillMetabase/2000_objects-8 1 4668489311 ns/op 2245232944 B/op 1399796 allocs/op // 4.7s BenchmarkRefillMetabase/5000_objects-8 1 11616198474 ns/op 5628608720 B/op 3457276 allocs/op // 11.6s ``` ### RefillMetabaseWorkersCountDefault = 500 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2305 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 2306998473 ns/op 1140297008 B/op 843014 allocs/op // 2.3s BenchmarkRefillMetabase/2000_objects-8 1 4664998201 ns/op 2255111088 B/op 1519356 allocs/op // 4.7s BenchmarkRefillMetabase/5000_objects-8 1 11438317463 ns/op 5641908880 B/op 3599110 allocs/op // 11.4s ``` ### RefillMetabaseWorkersCountDefault = 1000 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2355 ns/op 0 B/op 0 allocs/op // 0.0s BenchmarkRefillMetabase/1000_objects-8 1 2241713913 ns/op 1137704440 B/op 840555 allocs/op // 2.2s BenchmarkRefillMetabase/2000_objects-8 1 4681489790 ns/op 2266312320 B/op 1750873 allocs/op // 4.7s BenchmarkRefillMetabase/5000_objects-8 1 11469093646 ns/op 5658172016 B/op 3847801 allocs/op // 11.5s ``` ### RefillMetabaseWorkersCountDefault = 2000 ``` goos: linux goarch: amd64 pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkRefillMetabase/100_objects-8 1000000000 0.2429 ns/op 0 B/op 0 allocs/op BenchmarkRefillMetabase/1000_objects-8 1 2269358666 ns/op 1140359296 B/op 850544 allocs/op // 2.2s BenchmarkRefillMetabase/2000_objects-8 1 4497506710 ns/op 2264781384 B/op 1772200 allocs/op // 4.5s BenchmarkRefillMetabase/5000_objects-8 1 11692943181 ns/op 5665310944 B/op 3870899 allocs/op // 11.7s ``` Metrics dashboards: ![image](/attachments/dbe50f16-e643-4866-89d3-7d94dbf28d8a) ![image](/attachments/577f4887-c6ec-4e1e-b265-5171f96a8083)
image.png
211 KiB
image.png
210 KiB
image.png
181 KiB
image.png
dstepanov-yadro reviewed 2024-03-05 14:13:04 +00:00
pkg/local_object_storage/blobovnicza/iterate.go Outdated
@ -160,2 +160,3 @@
if !prm.withoutData {
elem.data = v
data := make([]byte, 0, len(v))
elem.data = append(data, v...)
dstepanov-yadro commented 2024-03-05 14:13:04 +00:00
Author
Member
Copy link

v is only valid for tx lifetime. So to use v in separate goroutine it is required to copy it.

`v` is only valid for `tx` lifetime. So to use `v` in separate goroutine it is required to copy it.
fyrchik commented 2024-03-05 14:58:24 +00:00
Owner
Copy link

bytes.Clone()?

`bytes.Clone()`?
dstepanov-yadro commented 2024-03-05 15:24:29 +00:00
Author
Member
Copy link

bytes.Clone() doesn't preallocate enough capacity.

`bytes.Clone()` doesn't preallocate enough capacity.
fyrchik commented 2024-03-05 18:39:37 +00:00
Owner
Copy link

It does, it may overallocate though, but it is the solution from the stdlib.
Don't mind to leave it as is.

It does, it may overallocate though, but it is the solution from the stdlib. Don't mind to leave it as is.
dstepanov-yadro commented 2024-03-06 07:09:39 +00:00
Author
Member
Copy link

fixed

fixed
fyrchik marked this conversation as resolved
dstepanov-yadro requested review from storage-core-committers 2024-03-05 14:21:44 +00:00
dstepanov-yadro requested review from storage-core-developers 2024-03-05 14:21:45 +00:00
acid-ant approved these changes 2024-03-05 14:52:48 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from 8727d55371 to 7e1b5b8c37 2024-03-06 07:09:27 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from 7e1b5b8c37 to 4c479b861d 2024-03-06 07:10:52 +00:00 Compare
acid-ant approved these changes 2024-03-06 07:27:39 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from d92a98923a to 0d91716e81 2024-03-11 13:41:46 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from 0d91716e81 to 78a96c8182 2024-03-12 07:43:34 +00:00 Compare
acid-ant approved these changes 2024-03-12 07:54:03 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from 387ec06718 to a122ef3cd9 2024-03-12 14:38:13 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from a122ef3cd9 to d4e315aba0 2024-03-12 14:53:14 +00:00 Compare
acid-ant reviewed 2024-03-12 15:12:22 +00:00
pkg/local_object_storage/shard/control.go Outdated
@ -187,2 +202,3 @@
obj := objectSDK.New()
withDiskUsage := true
duCtx, cancel := context.WithTimeout(ctx, 10*time.Minute) // if compute disk usage takes too long, then just skip this step
acid-ant commented 2024-03-12 15:12:22 +00:00
Member
Copy link

Why not to move timeout in config?

Why not to move timeout in config?
dstepanov-yadro commented 2024-03-12 15:18:57 +00:00
Author
Member
Copy link

can't imagine the situation when we will change this.

can't imagine the situation when we will change this.
acid-ant approved these changes 2024-03-12 15:19:16 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from d4e315aba0 to dfc2fe2ebf 2024-03-13 07:09:19 +00:00 Compare
aarifullin approved these changes 2024-03-13 10:36:19 +00:00
aarifullin left a comment
Member
Copy link

LGTM

LGTM
fyrchik commented 2024-04-05 13:37:11 +00:00
Owner
Copy link

Doesn't this close #1024?

Doesn't this close #1024?
dstepanov-yadro commented 2024-04-08 08:07:00 +00:00
Author
Member
Copy link

Doesn't this close #1024?

It does. But I prefer to close tasks manually with comments.

> Doesn't this close #1024? It does. But I prefer to close tasks manually with comments.
dstepanov-yadro force-pushed feat/resync_speedup from dfc2fe2ebf to 182ec28f1c 2024-04-08 08:08:37 +00:00 Compare
fyrchik requested changes 2024-04-08 08:46:46 +00:00
pkg/local_object_storage/blobstor/info.go Outdated
@ -18,0 +49,4 @@
for i := range b.storage {
path := b.storage[i].Storage.Path()
eg.Go(func() error {
return filepath.WalkDir(path, func(path string, d fs.DirEntry, _ error) error {
fyrchik commented 2024-04-08 08:44:43 +00:00
Owner
Copy link

This won't work for blobovniczas:

  1. They don't reclaim memory and the whole 40mb db could be empty.
  2. B-tree has some overhead for meta.

This becomes worse the more small objects we have.

Have you considered using naive but correct implementation? (traverse blobovniczas, maybe even count objects, not their size).

This won't work for blobovniczas: 1. They don't reclaim memory and the whole 40mb db could be empty. 2. B-tree has some overhead for meta. This becomes worse the more small objects we have. Have you considered using naive but correct implementation? (traverse blobovniczas, maybe even count objects, not their size).
dstepanov-yadro commented 2024-04-08 09:01:49 +00:00
Author
Member
Copy link

That's why the method is called DiskUsage, not DataSize.
Every blobovnicza stores data size, but to get this value it is required to open/close db. As it is used only for metrics, I think it is enough to calculate only disk size.

That's why the method is called `DiskUsage`, not `DataSize`. Every blobovnicza stores data size, but to get this value it is required to open/close db. As it is used only for metrics, I think it is enough to calculate only disk size.
fyrchik commented 2024-04-09 15:32:39 +00:00
Owner
Copy link

But then metrics won't reflect what we deem them reflect (progress of operation).

But then metrics won't reflect what we deem them reflect (progress of operation).
dstepanov-yadro commented 2024-04-10 11:27:10 +00:00
Author
Member
Copy link

Agree, fixed.
Now object count used for metrics, data size is hard to compute if compression is enabled.

Agree, fixed. Now object count used for metrics, data size is hard to compute if compression is enabled.
pkg/local_object_storage/shard/control.go Outdated
@ -187,2 +202,3 @@
obj := objectSDK.New()
withDiskUsage := true
duCtx, cancel := context.WithTimeout(ctx, 10*time.Minute) // if compute disk usage takes too long, then just skip this step
fyrchik commented 2024-04-08 08:35:56 +00:00
Owner
Copy link

It looks strange to me: we can wait for 10 minutes and still do not receive any metrics.
Ironically situations when disk usage calculation takes too long are also likely the ones where having metrics provides a huge QOL improvement. metabase resync duration has the order of tens of hours for big disks, 10 minute is nothing in comparison.

So why don't we allow ourselves hang here?

It looks strange to me: we can wait for 10 minutes and still do not receive any metrics. Ironically situations when disk usage calculation takes too long are also likely the ones where having metrics provides a huge QOL improvement. metabase resync duration has the order of _tens of hours_ for big disks, 10 minute is nothing in comparison. So why don't we allow ourselves hang here?
dstepanov-yadro commented 2024-04-08 09:04:59 +00:00
Author
Member
Copy link

Fixed.

Fixed.
fyrchik marked this conversation as resolved
dstepanov-yadro force-pushed feat/resync_speedup from 182ec28f1c to 0306fdc818 2024-04-08 09:04:52 +00:00 Compare
acid-ant approved these changes 2024-04-09 06:58:49 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from 0306fdc818 to 23fdee2e97 2024-04-09 07:19:18 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from 23fdee2e97 to b66ea2c713 2024-04-09 16:48:10 +00:00 Compare
dstepanov-yadro changed title from Speed up metabase refill to WIP: Speed up metabase refill 2024-04-09 16:48:30 +00:00
dstepanov-yadro force-pushed feat/resync_speedup from b66ea2c713 to 6e038936cf 2024-04-10 07:21:47 +00:00 Compare
dstepanov-yadro force-pushed feat/resync_speedup from 6e038936cf to 1b17258c04 2024-04-10 11:26:19 +00:00 Compare
dstepanov-yadro changed title from WIP: Speed up metabase refill to Speed up metabase refill 2024-04-10 11:30:16 +00:00
fyrchik approved these changes 2024-04-10 12:08:44 +00:00
fyrchik merged commit 1b17258c04 into master 2024-04-10 12:08:59 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
aarifullin
acid-ant
fyrchik
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
badger

Badger DB related issues

  
frostfs-adm

   
frostfs-cli

   
frostfs-ir

   
frostfs-lens

   
frostfs-node

   
good first issue

Good for newcomers

  
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#1027
No description provided.
Delete branch "dstepanov-yadro/frostfs-node:feat/resync_speedup"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?