TrueCloudLab/frostfs-node
20
1
Fork 27
Code Issues 100 Pull requests 4 Projects Releases 1 Activity Actions

[#370] Add tree service metrics #388

Merged
fyrchik merged 1 commit from ale64bit/frostfs-node:feature/370-metrics-tree-service into master 2023-05-26 13:39:14 +00:00
Conversation 0 Commits 1 Files changed 9 +117 -3
ale64bit commented 2023-05-24 07:06:28 +00:00
Member
Copy link

Signed-off-by: Alejandro Lopez a.lopez@yadro.com
Close #370

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com> Close #370
ale64bit requested review from storage-core-committers 2023-05-24 07:06:46 +00:00
ale64bit requested review from storage-core-developers 2023-05-24 07:06:47 +00:00
acid-ant approved these changes 2023-05-24 08:07:26 +00:00
dstepanov-yadro reviewed 2023-05-24 08:51:54 +00:00
pkg/services/tree/replicator.go Outdated
@ -143,3 +145,4 @@
zap.String("err", err.Error()),
zap.Stringer("cid", op.cid),
zap.String("treeID", op.treeID))
s.metrics.IncReplicateErrors()
dstepanov-yadro commented 2023-05-24 08:51:51 +00:00
Member
Copy link

Why failed replicate operation duration is not measured?

Why failed replicate operation duration is not measured?
ale64bit commented 2023-05-24 08:58:56 +00:00
Author
Member
Copy link

I was concerned that if some of them failed too quick, they would skew the distribution of those that succeeded.

I was concerned that if some of them failed too quick, they would skew the distribution of those that succeeded.
dstepanov-yadro commented 2023-05-24 13:56:42 +00:00
Member
Copy link

We can use labels.
There are best practices: https://prometheus.io/docs/practices/naming/#labels

Use labels to differentiate the characteristics of the thing that is being measured:

* api_http_requests_total - differentiate request types: operation="create|update|delete"

* api_request_duration_seconds - differentiate request stages: stage="extract|transform|load"
We can use labels. There are best practices: https://prometheus.io/docs/practices/naming/#labels ``` Use labels to differentiate the characteristics of the thing that is being measured: * api_http_requests_total - differentiate request types: operation="create|update|delete" * api_request_duration_seconds - differentiate request stages: stage="extract|transform|load" ```
ale64bit commented 2023-05-25 07:16:34 +00:00
Author
Member
Copy link

done. Thanks for the suggestion.

done. Thanks for the suggestion.
dstepanov-yadro marked this conversation as resolved
fyrchik reviewed 2023-05-24 10:28:47 +00:00
pkg/services/tree/options.go Outdated
@ -28,11 +28,14 @@ type cfg struct {
cnrSource ContainerSource
eaclSource container.EACLSource
forest pilorama.Forest
fyrchik commented 2023-05-24 10:22:30 +00:00
Owner
Copy link

unrelated to the commit.

unrelated to the commit.
ale64bit commented 2023-05-24 11:56:15 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
pkg/services/tree/options.go Outdated
@ -119,0 +122,4 @@
func WithMetrics(v MetricsRegister) Option {
return func(c *cfg) {
c.metrics = v
fyrchik commented 2023-05-24 10:23:13 +00:00
Owner
Copy link

What about sane dummy non-null default?

What about sane dummy non-null default?
ale64bit commented 2023-05-24 11:56:20 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
pkg/services/tree/replicator.go Outdated
@ -70,6 +70,7 @@ func (s *Service) replicationWorker(ctx context.Context) {
case <-s.closeCh:
return
case task := <-s.replicationTasks:
s.metrics.IncReplicateTasks()
fyrchik commented 2023-05-24 10:28:45 +00:00
Owner
Copy link

And here we measure each operation multiple times. Was in intended? Compare with AddReplicateDuration.
What about replacing that duration with ReplicateWaitTime, and using Count + Duration here?

And here we measure each operation multiple times. Was in intended? Compare with `AddReplicateDuration`. What about replacing that duration with `ReplicateWaitTime`, and using Count + Duration here?
ale64bit commented 2023-05-24 11:56:32 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
pkg/services/tree/replicator.go Outdated
@ -145,1 +147,4 @@
zap.String("treeID", op.treeID))
s.metrics.IncReplicateErrors()
} else {
s.metrics.AddReplicateDuration(time.Since(start))
fyrchik commented 2023-05-24 10:26:57 +00:00
Owner
Copy link

Just to be aware: replicate does 3 things:

  1. Signs the operation.
  2. Calculates the list op nodes to replicate to.
  3. Sends the request.

While this metric is useful (how much time do we wait in queue), can we also measure the time spent in replicationWorker?

Just to be aware: `replicate` does 3 things: 1. Signs the operation. 2. Calculates the list op nodes to replicate to. 3. Sends the request. While this metric _is_ useful (how much time do we wait in queue), can we also measure the time spent in `replicationWorker`?
ale64bit commented 2023-05-24 11:56:46 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
pkg/services/tree/sync.go Outdated
@ -365,6 +366,32 @@ func (s *Service) SynchronizeAll() error {
}
}
func (s *Service) sync(ctx context.Context) {
fyrchik commented 2023-05-24 10:24:04 +00:00
Owner
Copy link

Again, what about separate commit?

Again, what about separate commit?
ale64bit commented 2023-05-24 11:56:52 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
ale64bit force-pushed feature/370-metrics-tree-service from ee1aced7bd to 6d4047d00d 2023-05-24 11:55:59 +00:00 Compare
ale64bit force-pushed feature/370-metrics-tree-service from 6d4047d00d to 92f89d33b6 2023-05-25 07:16:08 +00:00 Compare
dstepanov-yadro reviewed 2023-05-25 13:10:23 +00:00
pkg/metrics/treeservice.go Outdated
@ -0,0 +46,4 @@
Name: "replicate_wait_duration_seconds",
Help: "Duration of overall waiting time for replication loops",
}, []string{treeServiceLabelSuccess}),
syncOps: newCounter(prometheus.CounterOpts{
dstepanov-yadro commented 2023-05-25 13:10:19 +00:00
Member
Copy link

@ale64bit @fyrchik Maybe it's not worth separating operations and errors, but using labels everywhere? In fact, there is an operation that ended with an error or without an error.

@ale64bit @fyrchik Maybe it's not worth separating operations and errors, but using labels everywhere? In fact, there is an operation that ended with an error or without an error.
ale64bit commented 2023-05-25 13:30:10 +00:00
Author
Member
Copy link

I didn't fully understand what you meant by "In fact, there is an operation that ended with an error or without an error."?

I didn't fully understand what you meant by *"In fact, there is an operation that ended with an error or without an error."*?
dstepanov-yadro commented 2023-05-25 13:40:25 +00:00
Member
Copy link

I'm sorry, I didn't make myself clear enough.
This code:

			case <-s.syncChan:

			s.metrics.IncSyncOps()

			cnrs, err := s.cfg.cnrSource.List()
			if err != nil {
				s.metrics.IncSyncErrors()
				return
			}

Operations and errors are counted separately. I think it should be a single counter with label separation.
If we need to display the number of success operations, then we should use such expression: irate(sync_operation[1m]) - irate(sync_operation_errors[1m]) instead of irate(sync_operation{success="true"}[1m])

I'm sorry, I didn't make myself clear enough. This code: ``` case <-s.syncChan: s.metrics.IncSyncOps() cnrs, err := s.cfg.cnrSource.List() if err != nil { s.metrics.IncSyncErrors() return } ``` Operations and errors are counted separately. I think it should be a single counter with label separation. If we need to display the number of success operations, then we should use such expression: `irate(sync_operation[1m]) - irate(sync_operation_errors[1m])` instead of `irate(sync_operation{success="true"}[1m])`
ale64bit commented 2023-05-25 13:52:27 +00:00
Author
Member
Copy link

You mean that we should just use a single HistogramVec for operations in general? (with a success label). Since it includes a count of the observations as well.

You mean that we should just use a single `HistogramVec` for operations in general? (with a `success` label). Since it includes a count of the observations as well.
dstepanov-yadro commented 2023-05-25 13:57:41 +00:00
Member
Copy link

yes

yes
ale64bit commented 2023-05-25 14:04:35 +00:00
Author
Member
Copy link

I agree it's better. Done.

I agree it's better. Done.
dstepanov-yadro marked this conversation as resolved
dstepanov-yadro reviewed 2023-05-25 13:41:04 +00:00
pkg/services/tree/sync.go Outdated
@ -382,2 +387,3 @@
s.metrics.IncSyncErrors()
span.End()
continue
return
dstepanov-yadro commented 2023-05-25 13:41:00 +00:00
Member
Copy link

why return?

why return?
ale64bit commented 2023-05-25 14:04:40 +00:00
Author
Member
Copy link

fixed

fixed
dstepanov-yadro marked this conversation as resolved
ale64bit force-pushed feature/370-metrics-tree-service from 92f89d33b6 to 87cc5877ef 2023-05-25 14:03:47 +00:00 Compare
dstepanov-yadro approved these changes 2023-05-25 14:28:54 +00:00
fyrchik approved these changes 2023-05-26 09:10:05 +00:00
pkg/metrics/node.go Outdated
@ -9,9 +12,10 @@ type NodeMetrics struct {
engineMetrics
stateMetrics
replicatorMetrics
epoch metric[prometheus.Gauge]
fyrchik commented 2023-05-26 09:07:59 +00:00
Owner
Copy link

Why did we move this line?

Why did we move this line?
ale64bit commented 2023-05-26 10:34:23 +00:00
Author
Member
Copy link

I moved it to clearly split the embedded fields from the others, including this one.

I moved it to clearly split the embedded fields from the others, including this one.
pkg/services/tree/replicator.go Outdated
@ -143,3 +148,4 @@
zap.String("err", err.Error()),
zap.Stringer("cid", op.cid),
zap.String("treeID", op.treeID))
s.metrics.AddReplicateWaitDuration(time.Since(start), false)
fyrchik commented 2023-05-26 09:09:34 +00:00
Owner
Copy link

Can we unify 3 lines with s.metrics.AddReplicateWaitDuration(time.Since(start), err == nil)?

Can we unify 3 lines with `s.metrics.AddReplicateWaitDuration(time.Since(start), err == nil)`?
ale64bit commented 2023-05-26 10:34:35 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
pkg/services/tree/sync.go Outdated
@ -386,10 +391,11 @@ func (s *Service) syncLoop(ctx context.Context) {
newMap, cnrsToSync := s.containersToSync(cnrs)
s.syncContainers(ctx, cnrsToSync)
fyrchik commented 2023-05-26 09:09:48 +00:00
Owner
Copy link

Why this change?

Why this change?
ale64bit commented 2023-05-26 10:34:55 +00:00
Author
Member
Copy link

the empty line seemed superfluous to me.

the empty line seemed superfluous to me.
pkg/services/tree/sync.go Outdated
@ -374,11 +375,15 @@ func (s *Service) syncLoop(ctx context.Context) {
return
case <-s.syncChan:
ctx, span := tracing.StartSpanFromContext(ctx, "TreeService.sync")
fyrchik commented 2023-05-26 09:09:53 +00:00
Owner
Copy link

ahh

ahh
ale64bit commented 2023-05-26 10:35:42 +00:00
Author
Member
Copy link

If by "ahh" you mean "I don't like this empty line", I removed it. I wish gofmt would eventually have a say on empty lines, to avoid wasting time discussing them.

If by "ahh" you mean "I don't like this empty line", I removed it. I wish `gofmt` would eventually have a say on empty lines, to avoid wasting time discussing them.
fyrchik commented 2023-05-26 10:48:28 +00:00
Owner
Copy link

It's not about an empty line per se, it's about unrelated changes in the commit.

It's not about an empty line per se, it's about unrelated changes in the commit.
ale64bit force-pushed feature/370-metrics-tree-service from 87cc5877ef to bfded3a547 2023-05-26 10:34:16 +00:00 Compare
fyrchik approved these changes 2023-05-26 13:39:04 +00:00
fyrchik merged commit bc34fee6a7 into master 2023-05-26 13:39:14 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
acid-ant
dstepanov-yadro
fyrchik
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
badger

Badger DB related issues

  
frostfs-adm

   
frostfs-cli

   
frostfs-ir

   
frostfs-lens

   
frostfs-node

   
good first issue

Good for newcomers

  
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#388
No description provided.
Delete branch "ale64bit/frostfs-node:feature/370-metrics-tree-service"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?