Please help with metric tunning

Grafana Time Series Panel
1

Hi there, I’m really new to grafana trying to implement some monitoring in our company.

The story is that we have a few backend servers processing some calls to other systems. We often release a new version of applications and need to see how it affects the overall performance. There is about 150 of this calls. I’ve made a dasboard which perfectly suits my needs, but problem is, it is desperatly slow, it does not even load if I have more than 10 rows on the dashboard (but I need all 150 of them). I’m sure it’s due to my logic expression, it’s just too difficult to calculate the moving average for last 14days I know, but I don’t know, how to make it simpler. Can you please help? The performance of grafana server is not bad at all (some new Xeon Quad Core…), version of Grafana 4.1.2, version of graphite is 0.9.13

My goal is to compare current time of the call with average of the call times for last 14 days.
The slow graphs are: “Calling time change” and “% Change”

Capture.PNG1916×796 156 KB

This is the expression for “Calling time change”:
alias(movingAverage(offset(divideSeries(scale(stats.timers.wscbe.OUT.BssFacadeBundleModelSoapServiceClientSessionManagementService.GetSessionDetail.OK.SOAP.mean,100),movingAverage(stats.timers.wscbe.OUT.BssFacadeBundleModelSoapServiceClientSessionManagementService.GetSessionDetail.OK.SOAP.mean, ‘14d’)),-100), ‘1hour’), ‘change’)

And this for “% Change”:
alias(offset(divideSeries(scale(stats.timers.wscbe.OUT.BssFacadeBundleModelSoapServiceClientSessionManagementService.GetSessionDetail.OK.SOAP.mean,100),movingAverage(stats.timers.wscbe.OUT.BssFacadeBundleModelSoapServiceClientSessionManagementService.GetSessionDetail.OK.SOAP.mean, ‘14d’)),-100), ‘change’)

2

As far as I can tell, these queries shouldn’t be that slow. Especially because you are querying a single series, and not doing a bunch of regex matches that might operate on thousands of individual series.

Some thoughts/questions that might help:

  1. How ‘slow’ is slow? A couple of seconds? 30 seconds? Longer?

  2. What resolution is your data? How much data are you sending? Are rollups configured properly? Is the server highly loaded? Things might need to be tuned.

  3. You might want to consider using Hosted Metrics on GrafanaCloud – it’s an extremely high performance Graphite-compatible backend, and we can deal with the tuning, scaling and maintenance for you.