I apologize for the clickbait-y title, but it does describe my problem.
I’m new to Grafana (but not to BI/DataViz tools) and I have to confess that my first experience with this tool has been disappointing.
My first task in a new job is to create a dashboard to show failures in a data ingestion pipeline. I have an AWS CloudWatch metric that indicates total number of messages and another metric that shows successfully ingested metrics. This should be easy-- I want to display
(total - success) / total * 100
This gives me the percentage of messages that failed ingestion.
Either I haven’t learned Grafana well enough, or Grafana makes it really hard to do this. I can use an Expression to do the calculation in one pass, but Expressions are still a beta feature… and Grafana doesn’t give me the ability to name this expression in the Panel, which is a major issue.
Alternatively, I can use a series of transforms, in which case I can rename the displayed values in the panel to display the value I want. This is a chore since I need to chain three expressions: one to get the delta between total and success, one to divide that by total, and a third to multiply by 100. Not an ideal solution, but it works.
Except… both approaches are giving me the wrong answer!
Check out the screenshot:
Here, you can see values for ‘Total’ and ‘Success’ in the top row. In the second row, the Transformation calculation shows the delta between the two is 34, which represents the number of failures.
On my calculator, 34 divided by 13,328,944 is 2.55E-6. According to Grafana, this expression evaluates to 0.000131. This incorrect result happens both in the Expression and the Transformation.
I clearly can’t publish a dashboard that doesn’t do math correctly. Am I doing something wrong here, or is Grafana broken?
Here’s the panel JSON:
{
"id": 2,
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 0
},
"type": "stat",
"title": "Panel Title",
"transformations": [
{
"id": "calculateField",
"options": {
"mode": "binary",
"reduce": {
"reducer": "sum"
},
"alias": "FailedCount",
"binary": {
"left": "Total",
"operator": "-",
"reducer": "sum",
"right": "Success"
}
}
},
{
"id": "calculateField",
"options": {
"mode": "binary",
"reduce": {
"reducer": "sum"
},
"alias": "FailedRatio",
"binary": {
"left": "FailedCount",
"operator": "/",
"reducer": "sum",
"right": "Total"
}
}
},
{
"id": "calculateField",
"options": {
"alias": "FailedPercent",
"binary": {
"left": "FailedRatio",
"operator": "*",
"reducer": "sum",
"right": "100"
},
"mode": "binary",
"reduce": {
"reducer": "sum"
}
}
}
],
"pluginVersion": "8.3.2",
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"mappings": [],
"color": {
"mode": "thresholds"
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"values": false,
"calcs": [
"sum"
],
"fields": ""
},
"orientation": "auto",
"textMode": "auto",
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto"
},
"targets": [
{
"queryMode": "Metrics",
"namespace": "AWS/IoT",
"metricName": "TopicMatch",
"expression": "",
"dimensions": {
"RuleName": "<redacted>"
},
"region": "default",
"id": "",
"alias": "Total",
"statistic": "SampleCount",
"period": "",
"metricQueryType": 0,
"metricEditorMode": 0,
"sqlExpression": "",
"datasource": {
"type": "cloudwatch",
"uid": "<redacted>"
},
"matchExact": false,
"refId": "A"
},
{
"queryMode": "Metrics",
"namespace": "AWS/IoT",
"metricName": "Success",
"expression": "",
"dimensions": {
"RuleName": "<redacted>"
},
"region": "default",
"id": "",
"alias": "Success",
"statistic": "SampleCount",
"period": "",
"metricQueryType": 0,
"metricEditorMode": 0,
"sqlExpression": "",
"datasource": {
"type": "cloudwatch",
"uid": "<redacted>"
},
"hide": false,
"matchExact": false,
"refId": "B"
},
{
"refId": "C",
"type": "math",
"datasource": {
"type": "__expr__",
"uid": "__expr__"
},
"hide": false,
"expression": "($A - $B) / $A * 100"
}
],
"datasource": null
}
Any guidance would be appreciated
EDIT: This is on Grafana 8.3.2.
.