How to create a dashboard from ElasticSearch logs

Grafana Elasticsearch
1

I have connected grafana to an elasticsearch data source. The goal is to create metric dashboards out of elastic search logs. I have two lucene queries in grafana. The first query retrieves all logs in the specified log path

log.file.path:\/data\/gitlab\/logs\/gitlab-rails\/release_job_logs.json AND fields.environment:production

The second query retrieves logs in the specified path that has a ‘success’ value in the json.job_status field

log.file.path:\/data\/gitlab\/logs\/gitlab-rails\/release_job_logs.json AND fields.environment:production AND json.job_status:success

I will like to create a dashboard that creates a percentage of the logs with a ‘success’ value in the json.job_status field.
So essentially if the first query gives me a count of 100 and the second query gives a count of 90 then the dashboard should display 90%. Which will mean 90% of all logs have a json.job_status:success

The image below shows what I have now from the two queries above. How do I get a percentage dashboard

Screenshot (1)
Screenshot (1)1920×1080 243 KB

2

Hi there,

Just go to transformation, then select Add field from calculation
Then select Binary Operator

Select the first value, then select the operation to /, last, select the second value.
Put the alias as Success Percentage

After this, you need to Override each of the alias, especially the unit
Put the unit as none for the first and second value, but the third value, select Percentage 0-1

And you will get the new Gauge as percentage.

image
image1463×365 50.4 KB

Regards,
Fadjar Tandabawana

3

Thanks for your answer.
Transform completely gives me a wrong percentage and also converts the first gauges into the percentage value. So say the percentage value is 10% ( which is wrong), the first gauges will be 10 and last gauge produced by transform will be 10%. I cant figure out why

4

Hi,

You need to go to the Override in the right pane and below after you scroll down.
Check the override per field and then add the override property, find the standard option - unit.
You can change the unit per override

Regards,
Fadjar Tandabawana

5

As you can see from the image, I have changed the overrides. I cant figure out what is wrong

Screenshot (4)
Screenshot (4)1920×1080 218 KB

6

Hi,

Please let me know the override config for the Success Percentage, also try to save it first then refresh.

Regards,
Fadjar Tandabawana

7

It looks like this,

        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "log.offset 1"
            },
            "properties": [
              {
                "id": "unit",
                "value": "none"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "log.offset 2"
            },
            "properties": [
              {
                "id": "unit",
                "value": "none"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Success Percentage"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percentunit"
              }
            ]
          }
        ]
8

Try to select the calculation to the Last*

Regards,
Fadjar Tandabawana

9

That gives me 100% . I dont mind changing the gauge to another visualization…say stat or piechart for example