DatasourceError

Grafana Prometheus
, ,
1

Hi all,
it is not clear to me the cause of the error “DatasourceError” comes to me from time to time and usually change in “RESOLVING” after a few minutes.

What is the cause of the error of this datasource in prometheus?

Thanks Alen

2

Hi @alencappelletti ,

Can you please share some more details e.g. screenshot of the error message, and maybe error trace from the log file /var/log/grafana/grafana.log

So that we can know a bit more but it more sounds like that either the server where prometheus is running (if on another machine) is having connectivity/latency issues. But maybe with the help of logs, we can see in more detail.

3

Hi, I’m using Grafana v9.0.0 and this problem is related to a Prometheus connection in a ALERT.
As you can see below is the datatime highlighted.

image
image702×221 17.9 KB

I found a solution with a patch on “Alert state if execution error” . In this way I have no false positive

image

I think is something related to connectivity/latency issues as you said, and usually error return normal few minutes after. I found this log in grafana… (databse locked??) naturally I use SQLITE as default… I read could be better with many alert use mySQL or Postgres, is it true?

today at 10:11:13 AMlogger=sqlstore t=2022-07-13T08:11:13.552596312Z level=error msg="Failed getting data source" err="database is locked" uid=PBFA97CFB590B2093 id=0 name= orgId=1
today at 10:11:15 AMlogger=sqlstore t=2022-07-13T08:11:15.217607109Z level=error msg="Failed getting data source" err="database is locked" uid=PBFA97CFB590B2093 id=0 name= orgId=1
today at 10:11:18 AMlogger=context traceID=00000000000000000000000000000000 t=2022-07-13T08:11:18.427188444Z level=error msg="Failed to look up user based on cookie" error="user token not found"

Thank you very much for your support
ALEN

4

Hi Alen,

Thanks for the logs, yes it appears that the problem source is coming from the database as it might be not in stable state (due to a crash or not able to close properly).

You can follow this community post where there seems to be a solution i.e. to clone the current database and then replace it with the existing one as it may solve the issue.

Regarding the other part of your question, yes can use other database flavors e.g. MariaDB/MySQL or Postgres as it is totally up to you.

For e.g. if you think you are more familiar with MySQL then you can use it. The only things come to play later if your database grows in size, then may have some performance issues and atleast I know for MySQL there are several performance analyzer tools e.g. MySQLTuner to assist user with some suggestions.

1 Like
5

Hi and thanks for your reply, but do you think that using other types of databases these lock problems could disappear?

Since for now I’ve inserted only a small number of ALERTs but they will surely grow.

ALEN

6

There is not a correct answer to that because database corruption or instablity may happen if the service was not able to shutdown correctly, but with using a DB which you are more familiar with may help to identify the problem more better and easier. e.g. enabling debug logging or slow query logs etc.

1 Like
7

I moved on postgres… because increasing the number of writes inevitably leads to these problems, obviously if one has 5 alerts or simple things or in any case there is little interaction SQLite is ok.