[SOLVED] Date/Time issues - Dashboard Alerts/Metadata appear to one hour ahead?

Grafana
1

Hi All,

Strange problem, which I’ve only noticed since upgrading to 4.4.1 on a test box.
I wanted to see if others noticed the same, or if it’s something I’ve missed, before I log a bug on github.

Situation:
Using a single CentOS 7 dev/test server, and my own workstation (Windows).
Installed influxDB and/or Prometheus (just tried both to rule out datasource issues in any way - unlikely as I thought it’d be).
Metrics are going in fine, and are displaying correctly on panels.

The first time I notice the problem is when I configured alerts inside Grafana.
The moment I saved and activated the alert, it says “Pending for in an hour”.
I realised this was a little odd and began to dig, thinking the time sources on the server must be out or something.
Both my workstation and the server are reporting that they are synced up and the correct times (BST timezone).

Despite this, the dashboards seem to believe they are an hour ahead, no matter what.
Each time the alert status changes, it says “for in an hour”.

If I check the dashboard metadata, it too says the last modified time is an hour ahead of the actual time.
Panels are plotting at the correct times on histrogram X axis.

I’ve tried changing the dashboards timezone to UTC, local browser and default (presumably server time?) - None of which change the situation.
Checked through configuration, can’t see any timezone settings towards Grafana.

Am I missing something? Has anyone else noted this? Is there anything I can do, before/aside from logging a bug?

Many thanks!

2

strange, what database are you using? sqlite or mysql (for grafanas database)

1 Like
3

hi @torkel, thanks for the reply.

I haven’t explicitly configured a database, so I am presuming sqlite.

Come to think of it, not sure if this is relevant, but I normally install via yum repo however I was being a little hasty and just used the direct aws install method.

As I say, that could be irrelevant but I am just calling out anything that comes to mind.

4

I’ve just rebuilt it on a completely separate system completely.

Different workstation and server (still Windows/CentOS 7 respectively).

Able to replicate it completely: “ALERTING for in an hour”, despite that the graph panel correctly shows the marker for when it began alerting at the correct time.

Again, I used the main download instead of repo but doubt it makes a difference as I presume it’s just the same binary.

Server/Client times are same, metrics are stored with UTC timestamps.

It is using sqlite and if I open the grafana.db using sqlite3 and do SELECT CURRENT_TIMESTAMP; I am given the UTC timestamp.

I’m based in the UK (BST timezone), so it’s really puzzling me why Grafana is picking up anything to do with a timezone one hour in front for alerting and metadata (last modified dashboard etc).

If required, I can prepare some screen grabs to share?

5

@torkel - Just swapped out sqlite for mysql… Working fine now.
Do you have any thoughts as to why sqlite might be producing these kind of results?

Edit:
Grabbed this from the sqlite database

sqlite> select * from alert; 1|0|1|1|1|Panel Title alert||alerting|{"conditions":[{"evaluator":{"params":[7000],"type":"gt"},"operator":{"type":"and"},"query":{"datasourceId":1,"model":{"expr":"my_python_process","format":"time_series","intervalFactor":2,"metric":"my_python_process","refId":"A","step":2},"params":["A","10s","now"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"executionErrorState":"alerting","frequency":"10s","handler":1,"name":"Panel Title alert","noDataState":"no_data","notifications":[]}|10|1||0| |{"evalMatches":[{"value":11180,"metric":"my_python_process{instance=\"localhost:8000\", job=\"my_first_python\"}","tags":{"__name__":"my_python_process","instance":"localhost:8000","job":"my_first_python"}}]}||2017-07-22 23:23:01|2|2017-07-23 01:22:40|2017-07-23 01:22:40 sqlite> select new_state_date from alert; 2017-07-22 23:23:01

new_state_date looks correctly stored in UTC, but created/modified dates are UTC+2.

6

What timezone is the grafana server set to? Think sqlite is more sensitive to ENV timezone

7

It is set to British Summer Time.

8

@torkel - I attempted changing the operating system timezone to simply UTC.

I re-saved the dashboard and checked the Metadata.
Created date is now UTC+1.

Seems to be taking into account somewhere the current timezone but adding an hour?

The alert status just updated and Grafana still shows “OK for in an hour.”

9

Anyone able to offer any further advice to this?
I find it very strange that this would be a SQLite root cause - It used to work fine for me.

Anyone able to replicate it themselves or seen similar?

10

I’ve tested using a different distribution of linux (Ubuntu).
Same results when using SQLite.

Should I look to raise this as a bug via issues on Github?

(Using 4.4.3 Grafana)

11

we have fixed this in latest Grafana (nightly builds), will be included in 4.4.4 patch release

12

Thanks @torkel, just out of interest - what was the problem?

Edit: I presume this issue https://github.com/grafana/grafana/issues/8996