TrueCloudLab/frostfs-qos
14
0
Fork 6
Code Issues 1 Pull requests Projects Releases Packages Activity Actions

mclock: Fix timer-based scheduling #12

Merged
fyrchik merged 3 commits from dstepanov-yadro/frostfs-qos:fix/mclock_timer_scheduling into master 2025-03-21 13:41:31 +00:00
Conversation 1 Commits 3 Files changed 4 +69 -34
dstepanov-yadro commented 2025-03-19 13:00:23 +00:00
Member
Copy link

Let's assume that there are two requests in the queue with execution time t1 and t2.
The timer is set to t1. The timer is triggered, schedules the t1 request,
calculates the time for the next timer t2 to be triggered.
But it doesn't schedules timer to this time because of the
q.timeBasedScheduleTs > nextTs check.

Let's assume that there are two requests in the queue with execution time t1 and t2. The timer is set to t1. The timer is triggered, schedules the t1 request, calculates the time for the next timer t2 to be triggered. But it doesn't schedules timer to this time because of the `q.timeBasedScheduleTs > nextTs` check.
dstepanov-yadro added 1 commit 2025-03-19 13:00:23 +00:00
[#9999] mclock: Fix timer-based scheduling
All checks were successful
DCO action / DCO (pull_request) Successful in 25s
Details
Tests and linters / Run gofumpt (pull_request) Successful in 28s
Details
Tests and linters / Tests (pull_request) Successful in 43s
Details
Vulncheck / Vulncheck (pull_request) Successful in 40s
Details
Tests and linters / Tests with -race (pull_request) Successful in 50s
Details
Tests and linters / Staticcheck (pull_request) Successful in 59s
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m28s
Details
Tests and linters / Lint (pull_request) Successful in 1m27s
Details
Tests and linters / gopls check (pull_request) Successful in 1m48s
Details
480b9aee3d 
Let's assume that there are two requests in the queue with execution time t1 and t2.
The timer is set to t1. The timer is triggered, schedules the t1 request,
calculates the time for the next timer t2 to be triggered.
But it doesn't schedules timer to this time because of the
`q.timeBasedScheduleTs > nextTs` check.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
requested review from fyrchik 2025-03-19 13:00:23 +00:00
dstepanov-yadro force-pushed fix/mclock_timer_scheduling from 480b9aee3d to f546741c3a 2025-03-19 13:01:13 +00:00 Compare
requested reviews from storage-core-committers, storage-core-developers 2025-03-19 13:01:29 +00:00
dstepanov-yadro reviewed 2025-03-19 13:07:37 +00:00
dstepanov-yadro changed title from mclock: Fix timer-based scheduling to WIP: mclock: Fix timer-based scheduling 2025-03-19 13:20:50 +00:00
fyrchik approved these changes 2025-03-19 13:25:19 +00:00
fyrchik left a comment
Owner
Copy link

How have you caught this bug?

How have you caught this bug?
scheduling/mclock.go Outdated
@ -300,3 +295,1 @@
q.scheduleRequest()
})
q.timeBasedScheduleTs = nextTs
if nextTs == math.MaxFloat64 {
fyrchik commented 2025-03-19 13:24:13 +00:00
Owner
Copy link

Comparing float64 with == rubs me the wrong way, do we have any alternative?

Comparing `float64` with `==` rubs me the wrong way, do we have any alternative?
dstepanov-yadro commented 2025-03-19 14:12:38 +00:00
Author
Member
Copy link

fixed

fixed
dstepanov-yadro force-pushed fix/mclock_timer_scheduling from f546741c3a to 1ebed20b9d 2025-03-19 14:02:17 +00:00 Compare
dstepanov-yadro force-pushed fix/mclock_timer_scheduling from 1ebed20b9d to 346752477b 2025-03-19 14:12:07 +00:00 Compare
dstepanov-yadro commented 2025-03-19 14:12:29 +00:00
Author
Member
Copy link

@fyrchik wrote in #12 (comment):

How have you caught this bug?

Hardware testing

@fyrchik wrote in https://git.frostfs.info/TrueCloudLab/frostfs-qos/pulls/12#issuecomment-70762: > How have you caught this bug? Hardware testing
dstepanov-yadro reviewed 2025-03-19 14:14:49 +00:00
scheduling/clock.go
@ -63,0 +59,4 @@
if currentTask != nil {
c.wg.Add(1)
f := currentTask
go func() {
dstepanov-yadro commented 2025-03-19 14:14:49 +00:00
Author
Member
Copy link

Run f() on separate goroutine to not to stuck on channel push.

Run f() on separate goroutine to not to stuck on channel push.
dstepanov-yadro added 1 commit 2025-03-20 07:35:33 +00:00
[#12] mclock: Fix deadlock caused by mclock.Close
All checks were successful
DCO action / DCO (pull_request) Successful in 21s
Details
Vulncheck / Vulncheck (pull_request) Successful in 42s
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m14s
Details
Tests and linters / Lint (pull_request) Successful in 1m14s
Details
Tests and linters / Run gofumpt (pull_request) Successful in 1m10s
Details
Tests and linters / Staticcheck (pull_request) Successful in 1m17s
Details
Tests and linters / Tests (pull_request) Successful in 1m6s
Details
Tests and linters / Tests with -race (pull_request) Successful in 1m22s
Details
Tests and linters / gopls check (pull_request) Successful in 1m29s
Details
1ca213ee7c 
Deadlock scenario:
- mclock closed by `Close` method, it locks mutex and calls `clock.close`
- clock starts `scheduleRequest` goroutine, it tries to lock mutex
- `clock.Close` waits for all goroutines

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
dstepanov-yadro added 1 commit 2025-03-20 14:25:05 +00:00
[#12] grpc: Fix method name
All checks were successful
DCO action / DCO (pull_request) Successful in 27s
Details
Vulncheck / Vulncheck (pull_request) Successful in 40s
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m17s
Details
Tests and linters / Run gofumpt (pull_request) Successful in 1m15s
Details
Tests and linters / Tests (pull_request) Successful in 1m21s
Details
Tests and linters / Staticcheck (pull_request) Successful in 1m21s
Details
Tests and linters / Tests with -race (pull_request) Successful in 1m22s
Details
Tests and linters / Lint (pull_request) Successful in 1m25s
Details
Tests and linters / gopls check (pull_request) Successful in 1m25s
Details
Vulncheck / Vulncheck (push) Successful in 48s
Details
Tests and linters / Run gofumpt (push) Successful in 1m11s
Details
Tests and linters / Staticcheck (push) Successful in 1m16s
Details
Tests and linters / Lint (push) Successful in 1m22s
Details
Pre-commit hooks / Pre-commit (push) Successful in 1m34s
Details
Tests and linters / Tests (push) Successful in 1m43s
Details
Tests and linters / Tests with -race (push) Successful in 1m45s
Details
Tests and linters / gopls check (push) Successful in 1m46s
Details
32079ad7c2 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
dstepanov-yadro changed title from WIP: mclock: Fix timer-based scheduling to mclock: Fix timer-based scheduling 2025-03-21 07:34:23 +00:00
requested reviews from fyrchik, storage-core-committers, storage-core-developers 2025-03-21 07:34:32 +00:00
aarifullin reviewed 2025-03-21 11:27:54 +00:00
scheduling/clock.go
@ -68,0 +73,4 @@
if s.ts >= currentTs {
// current timer will fire earlier
// so next scheduleRequest will push new schedule event
continue
aarifullin commented 2025-03-21 11:27:54 +00:00
Member
Copy link

But what does happen to the scheduled task? It's gone? Will it be rescheduled?

But what does happen to the scheduled task? It's gone? Will it be rescheduled?
dstepanov-yadro commented 2025-03-21 12:23:33 +00:00
Author
Member
Copy link

current timer will fire and call scheduleRequest, and scheduleRequest will set timer again on s.ts

current timer will fire and call `scheduleRequest`, and `scheduleRequest` will set timer again on `s.ts`
aarifullin marked this conversation as resolved
aarifullin approved these changes 2025-03-21 13:09:03 +00:00
fyrchik merged commit 32079ad7c2 into master 2025-03-21 13:41:31 +00:00
fyrchik referenced this pull request from a commit 2025-03-21 13:41:32 +00:00
[#12] grpc: Fix method name
Sign in to join this conversation.
Reviewers
No reviewers
fyrchik
aarifullin
TrueCloudLab/storage-core-committers
Labels
Clear labels
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-qos#12
No description provided.
Delete branch "dstepanov-yadro/frostfs-qos:fix/mclock_timer_scheduling"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?