Upgrade 4.2 issue ES 5.2

Grafana Elasticsearch
1

Hello, is anyone else running ES 5.2 had an issue after 4.2 upgrade?

After grafana upgrade all looks fine, until editing any chart query at which point the lucene query seems to break and the chart displays every single JSON key .

Have I had a failed upgrade or is this a genera issue?

Cheers,

Simon

2

We have had no reports of any breaking change yet.

3

Thanks for your reply, Very strange, I don’t appear to be able to edit any query, The charts look great until I attempt to edit the query.

I’ll snapshot back to previous release and look at it again later.

Best Regards
Simon

4

If you do please include query sent to ES and response. And any js errors (chrome dev tools console).

5

I am seeing this also, I reverted back to 4.1 and it works fine.

Not sure how to get grafana to show the queries it’s sending to elasticsearch so I can see
what broke when using 4.2 with es5.2

6

here is a guide for how to troubleshoot / investigate query issues (and see raw query sent and response):

https://localhost:3000/t/how-to-troubleshoot-metric-query-issues/50

7

Thanks for that, I’ll find some time next week to debug

8

I’ve found the problem, it’s Min Doc Count - which I think is new?

After upgrade Min Doc Count is 0 which seems to cause everything to be matched in a terms group by

If I set it to 1, all is well.

Thanks for your help.

9

I think you should consider setting default min_doc_count from default 0 to 1

By using 0 you’re overriding the ES default of 1 and causing terms to return that don’t match the query, and it breaks charts from previous release

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

Setting min_doc_count=0 will also return buckets for terms that didn’t match any hit. However, some of the returned terms which have a document count of zero might only belong to deleted documents or documents from other types, so there is no warranty that a match_all query would find a positive document count for those terms.

10

The problem with min_doc_count 1 is that it makes it impossible for Grafana to identify regions of missing data, there can be no gaps in the graphs in this case, so if you service goes down or stops sending data it’s going to be harder to see.

Buckets returned when min doc count is 0 are null, so should not have any impact (other than causing gaps in series where there is no data)

11

I see your point. It may be my particular case, where I do a query like this to get my NGINX data:

state.plugin_service_description:NGINX AND pfdta.metric_label:Active

Then do a terms group by on hosts to show all the hosts, then group by date histogram; it returns a load of hosts that are not NGINX servers with a zero doc count.

But tbh, its not much hassle to add a min_doc_count of 1.

Great product btw, streets ahead of Kibana for usability.

1 Like
12

So if the bucket is null, how do we keep the unwanted parts from displaying?

ie, if I wanted:

hostname: foo

I only want to see hostname: foo buckets, and nothing else.

13

Set min_doc_count to 1

14

so create a group by, and in it set the set the ‘Min Doc Count’ to 1 is what is needed.

and I can verify this did work; BUT, you have to set the min doc count on all the group-by’s. Setting in only on one group-by removes the empty buckets from that group by.

15

Yes, but whatever you order by, set min doc count to 1