Support prometheus histogram observation in pool metrics #245

New issue

Open

opened 2024-07-29 08:10:57 +00:00 by alexvanin · 0 comments

alexvanin commented

2024-07-29 08:10:57 +00:00

Owner

Prometheus supports histogram metrics that allow to look at distribution of observing value.

histogram = prometheus.NewHistogram(
	prometheus.HistogramOpts{
	Name:      "histogram",
})
...
histogram.Observe(float64(value))

This is heavily adopted in frostfs-node to monitor duration of storage engine operations:

func (b *blobovnicza) BlobobvnizcaTreeMethodDuration(shardID, path string, method string, d time.Duration, success bool, withStorageID NullBool) {
	b.treeReqDuration.With(prometheus.Labels{
		shardIDLabel:       shardID,
		pathLabel:          path,
		successLabel:       strconv.FormatBool(success),
		methodLabel:        method,
		withStorageIDLabel: withStorageID.String(),
	}).Observe(d.Seconds())
}

It would be nice to provide a way to produce duration histrogram metric with object and tree pools.

Describe the solution you'd like

Current statistic collecting approach is based on storing two atomic values for request counter and total duration. Client can provide a callback function, which is triggered when atomic values are upgraded.

func (c *clientWrapper) incRequests(elapsed time.Duration, method MethodIndex) {
...
	if c.statCallbackFunc != nil {
		c.statCallbackFunc(elapsed, method)
	}
...
}

Attention!
This is going to produce performance penalty, so we have to be careful by adding this.
Estimate performance penalty and consider mutual exclusive metric collection (avg or histogram)

Describe alternatives you've considered

Pool manages buckets by itself and works as prometheus metric provider.

## Is your feature request related to a problem? Please describe. Prometheus supports [histogram](https://prometheus.io/docs/concepts/metric_types/#histogram) metrics that allow to look at distribution of observing value. ```go histogram = prometheus.NewHistogram( prometheus.HistogramOpts{ Name: "histogram", }) ... histogram.Observe(float64(value)) ``` This is heavily adopted in frostfs-node to monitor duration of storage engine operations: ```go func (b *blobovnicza) BlobobvnizcaTreeMethodDuration(shardID, path string, method string, d time.Duration, success bool, withStorageID NullBool) { b.treeReqDuration.With(prometheus.Labels{ shardIDLabel: shardID, pathLabel: path, successLabel: strconv.FormatBool(success), methodLabel: method, withStorageIDLabel: withStorageID.String(), }).Observe(d.Seconds()) } ``` It would be nice to provide a way to produce duration histrogram metric with object and tree pools. ## Describe the solution you'd like Current statistic collecting approach is based on storing two atomic values for request counter and total duration. Client can provide a callback function, which is triggered when atomic values are upgraded. ```go func (c *clientWrapper) incRequests(elapsed time.Duration, method MethodIndex) { ... if c.statCallbackFunc != nil { c.statCallbackFunc(elapsed, method) } ... } ``` > **Attention!** > This is going to produce performance penalty, so we have to be careful by adding this. > Estimate performance penalty and consider mutual exclusive metric collection (avg or histogram) ## Describe alternatives you've considered Pool manages buckets by itself and works as prometheus metric provider.