How to instrument your backend data source

Grafana Plugin Development Share & showcase
1

In my previous post on how to display custom stats for your data source queries, I explained how you can return metrics about the query back to the user. This can be useful to dashboard authors who want to troubleshoot or optimize their queries. As an administrator however, you might want to understand how the data source performs over time, across dashboards. In this post, I’ll show you how you can instrument your backend data source using Prometheus.

First off, Grafana already exposes high-level metrics for your backend plugin under the /metrics endpoint, such as how many queries have been made:

grafana_plugin_request_duration_milliseconds_count{endpoint="queryData",plugin_id="myorg-datasource"} 13

With this, administrators can already configure their Prometheus instance to scrape /metrics to better understand how often data is requested from your data source. But for more complex data sources, you might want to expose more detailed metrics.

The Grafana SDK for Go lets you expose custom Prometheus metrics for your plugin. Here’s how you do it.

First, run the following command in the plugin directory to install the Prometheus client package for your plugin:

go get github.com/prometheus/client_golang

Next, register a counter for the number of queries for a given query type by adding the following code to your backend plugin:

var queriesTotal *prometheus.CounterVec

func init() {
	queriesTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name:      "query",
			Help:      "Total number of queries.",
			Namespace: "myorg_datasource",
		},
		[]string{"query_type"},
	)

	prometheus.MustRegister(queriesTotal)
}

Finally, to increment the the counter, add the following code in your query handler:

queriesTotal.WithLabelValues("logs").Inc()

Make some queries to increment the counter and then open /api/plugins/myorg-datasource/metrics in your browser to see the metrics. In addition to your custom metrics, Grafana also exposes metrics for the Go application itself.

Since the plugin metrics are exposed on a different endpoint, the administrator needs to configure their Prometheus instance to scrape both /metrics and /api/plugins/myorg-datasource/metrics.

For more information on how to use the Prometheus client package, check out Instrumenting a Go application for Prometheus.

2

Since Grafana v8.5.0 there’s a new metrics endpoint for backend plugins introduced that functions similar to Grafana’s existing /metrics endpoint. See Internal Grafana metrics | Grafana documentation for more information.

1 Like