Help with metric aggregation

Grafana Elasticsearch
1

Hi there, I’m working on a tutorial on how to display Spring Cloud Stream metrics using grafana.

I’m sending the metrics to an elasticsearch index that has the following mapping:

"properties" : {
    "name" : {"type" : "keyword","index" : "true", "ignore_above": 1024},
    "group" : {"type" : "keyword", "index" : "true", "ignore_above": 1024},
    "instanceIndex" : {"type" : "integer"},
    "@timestamp" : {"type" : "date"},
    "metrics" : {"type" : "nested"}

  }

And so far the dashboard support for templating is amazing. I’m using the group as the first option, and then the query drill down by filtering by group and grouping by name and instanceIndex.

The only side effect I noticed is that the series names are being shown as follows:

log 0, log 0 1, log 0 2, twitter 0, splitter 0

I get that 0 should represent the instanceIndex, but in case of the log metrics, which has 3 series, why it isn’t log 0, log 1, log 2?

Am I missing something here?

I also considered just grouping by name and having: log, twitter, splitter, but I could not find an easy way to drill down to a collection of apps to then split the panel by instance

Thank you

2

Hard to say without the actual response from your Elasticsearch for your query. Take a look at this issue for how to investigate the response from ES:

https://localhost:3000/t/how-to-troubleshoot-metric-query-issues/50

Also something that would help is if you let us know what your query Alias is (if left to auto or if you use an alias pattern)

3

Thank you for the quick reply. Helped a lot finding about the query that is being generated. I’m not using any alias at the moment. I tried using {{field name}} {{field instanceIndex}}, but for the ones with instanceIndex =0 I was getting undefined something like log undefined log 1 log 2 :frowning:

Here’s the query

{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "gte": "1489583525599",
              "lte": "1489669925599",
              "format": "epoch_millis"
            }
          }
        },
        {
          "query_string": {
            "analyze_wildcard": true,
            "query": "group:twitterStream AND name:(\"log\" OR \"splitter\" OR \"twitterstream\")"
          }
        }
      ]
    }
  },
  "aggs": {
    "3": {
      "terms": {
        "field": "name",
        "size": 10,
        "order": {
          "_term": "asc"
        }
      },
      "aggs": {
        "4": {
          "terms": {
            "field": "instanceIndex",
            "size": 10,
            "order": {
              "_term": "asc"
            }
          },
          "aggs": {
            "2": {
              "date_histogram": {
                "interval": "30s",
                "field": "@timestamp",
                "min_doc_count": 0,
                "extended_bounds": {
                  "min": "1489583525599",
                  "max": "1489669925599"
                },
                "format": "epoch_millis"
              },
              "aggs": {
                "1": {
                  "avg": {
                    "field": "metrics.integration.channel.input.send"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

And right now I’m banging my head against the wall, out of nothing ES is now returning null for all aggregations. I had this exact same query working yesterday. I don’t know if I touched something that could have changed that.

Regards

4

Hum, it appears one need to set a nested path since this is a nested property. I don’t know how it worked in first place. I manually run a slightly modified query and it worked.

{
  "size": 0,
  "query": {
    "match_all": {}
  },
  "aggs": {
    "metrics": {
      "nested": {
        "path": "metrics"
      },
      "aggs": {
        "average": {
          "avg": {
            "field": "metrics.integration.channel.input.send"
          }
        }
      }
    }
  }
}