Graphing Queries from Multiple Prometheus Instances

Grafana Time Series Panel
1

I have a simple Graph Panel that I am attempting to display query results from two different prometheus instances (A+B). The queries both execute successfully, but the Graph Panel appears to only display the results of the last query to return. Sometimes I will see the data appear for the first query a short period of time (until the second query returns).

Both prometheus instances are scraping and generating similar data, but for different servers in different datacenters. As such, the metric names and queries are identical, but the tuples returned have different labels / values. There is no way for us to federate the two Prometheus instances at this time.

Is there anything I can do to graph the results from both datasources to the same panel?

Example:

image
image1146×400 43.7 KB

1 Like
2

Also, this is for Grafana 4.5.2.

3

strange, when I test I always get results from both queries (and the graph shows both). Are you sure both queries return data?

The mixed data source will wait for the two queries completes and concat the series into one series list.

4

Let me go back at take a second look.

It looked like both queries were returning data, but I may have been getting responses to backed-up requests (due to some network issues in our environment).

5

Finally had a chance to go back and look: It appears that the user’s prometheus query was specifying too small a time range for the rate() function which was causing samples to appear disappear.

Thanks!