We push data into a elasticsearch DB using Python. The timestamps we push are timezone-aware datetime objects with the timezone set to UTC.
I inspected the query from Grafana for a selected time range of “2023-06-05 06:00:00” to “2023-06-06 02:00:00”. The queried epoch_millis “gte”:1685944800000 and “lte”:1686016800000 match the requested time range.
The data returned from the elasticsearch database is as expected and has the correct timestamps in UTC the first one being “2023-06-06T00:55:17.358420+00:00” so about 1am on the 6th of June UTC.
But in the graph the data is placed beginning from 15:00:00 on the 5th of June which is an offset of -10h. This suggests that Grafana is interpreting the timestamps from elasticsearch as local time (my timezone is UTC+10:00) and converts them to UTC by subtracting 10h before displaying them.
The timezone in the Grafana time range selector is set to UTC and the time series graph X-axis timezone is set to Default (but produces the same result if set to UTC).
This time mismatch between data and graph axis causes other problems. For example, I can’t zoom in on the data using the mouse on the graph. If I zoom closer as shown in the image above, I get an empty plot with a message saying “No data” because the queried time range (around 15:00) is incorrect. Or if I zoom in using the time range selector, I get an empty plot with a message saying “Data ouside time range” because the plot X-axis range is now around 01:00 on the 6th but Grafana wants to place the data at 15:00 on the 5th.
My Question:
What’s going on here and how can I fix this so that my data is plotted correctly? Is this a Grafana bug?
Even more strange: The "Table view" in the "Edit panel" area shows the timestamps with an offset of +10h (first timestamp being 2023-06-06 10:55:17) as if it interprets the data as UTC and converts it to local time.
That is by design. Grafana needs the timestamp in your datastore to be UTC then it displays it as your local time.
do you want it to be displayed as UTC?
The timestamp in our datastore (elasticsearch) is in UTC. From my experience with a different datastore (TimescaleDB/PostgreSQL) Grafana displays it in accordance to the timezone setting in the time range selector.
Even if it is as you describe, then Grafana would add 10h to the timestamp and display a data point from “2023-06-06T00:55:17.358420 UTC” in the graph at “2023-06-06T10:55:17.358420” as I’m in the timezone UTC+10:00.
But it does the opposite, it subtracts 10h and displays the data point in the graph at “2023-06-05T14:55:17.358420”. That’s why I think it interprets the timestamp from the datastore as my local time and displays it as UTC (that’s what I set in the time range selector) in the graph.
Please post a screen shot of the time picker
Also please share the line of code where you push this date via python?
Here is a screenshot of the time picker
This is a screenshot of a debug session running the code which is about to push the data into the elasticsearch DB. This happens in line 635 (the current code line). The dictionary with contains the data to be pushed is called lhk_doc. You can see it in the variables watch window. The timestamp (key = “datetime”) is shown there as well and you can see that it is a timezone-aware datetime with tzinfo = UTC.
You have you time picker set to UTC so it will be -10h. Are you in UTC? If not set it to your default local browser time like mine is PDT
I finally figured it out.
I changed the Grafana timezone setting under Configuration / Preferences to UTC (it was set to browser time) and now everything works as intended.
I can set the timezone in which I want to view the data in using the time picker and I can zoom into the graphs as I used to.
So I think Grafana doesn’t honor the time zone of the timestamps from the data store (maybe because timestamps not always have the time zone information). Instead it interprets the timestamps using the default time zone setting. As this was “browser time”, it interpreted the timestamps as UTC+10:00 and then converted it to UTC (this was what I had set in the time picker) by subtracting 10h.
1 Like
