Hi everyone,
I’m a newbie with database and visualization tool. Grafana looks perfect for my purpose: Monitoring power consummation of a house.
But I have the necessity to filter my raw data before displaying it and before compute some metrics on my dashboard. By filtering I mean signal frequency filtering such as low-pass/High-Pass filter.
I need to record raw data on my Database and the idea to store raw data and filtered data do not please me.
I have the necessity to filter my raw data before displaying it and before
compute some metrics on my dashboard. By filtering I mean signal frequency
filtering such as low-pass/High-Pass filter.
I think you need to be rather more specific about what that filtering looks
like. For example, what mathematical / statistical functions would you need
to apply to the raw data?
I need to record raw data on my Database and the idea to store raw data and
filtered data do not please me.
How about storing raw data in one database, having a processing function which
then saves processed data in another database, and then pointing Grafana at
the processed one?
Is it a way to accomplish what I want. Thank you.
Maybe give us some examples of your raw data, and what you would want it
processed into.
Thanks for answering. Maybe I’ll use a 2nd order Butterworth low pass filter. The difference equation is : y(k) = A * x(k-2) + B * x(k-1) + C * x(k) + D * y(k- 2) + E * y(k- 1),
Where ‘y’ is the ourput, ‘x’ the input, ‘A,B,C,D,E’ the filter constant coefficient and ‘k’ the current sample.
The data I want to process is time series curve with have some noise, especially spiky noise (i.e. none previsible spike) : like this for exemple https://i.stack.imgur.com/qsV5D.png
Thanks for answering. Maybe I’ll use a 2nd order Butterworth low pass
filter. The difference equation is:
y(k) = A * x(k-2) + B * x(k-1) + C * x(k) + D * y(k- 2) + E * y(k- 1),
Where ‘y’ is the output, ‘x’ the input, ‘A,B,C,D,E’ the filter constant
coefficient and ‘k’ the current sample.
Given that information, and what you said in your original posting, I think a
bit of clarification might help you.
Grafana itself is not a data manipulation tool, and it is certainly not a
statistical or signal processing analysis tool.
Grafana is a data visualisation tool, and the data is stored in any of a wide
variety of back-end data stores.
You can choose whichever you want and Grafana will work with it, and the data
processing functions which are available to you depend on what that data store
can do, more than it depends on anything in Grafana.
As it says on the page linked above “Each data source has a specific Query
Editor that is customized for the features and capabilities that the
particular data source exposes”.
Personally I don’t know whether any of those back-end data stores can do
Butterworth filtering functions - it’s possible, but you’d need to investigate
to find out.
However, there’s an important aspect of the way Grafana fetches data from the
back-end store - it will make a database query for every data series you want
to display on your dashboard panels, at every interval you specify they should
be updated. Therefore calculating something like a Butterworth filter for
every lookup would be a very computationally expensive process, and it would
be far better in my opinion to pre-process your raw data through such a
filter (which presumably only needs to be done once for each new sample x(k))
and store the processed results in a lightweight data store which Grafana can
then visualise for you.
The data I want to process is time series curve with have some noise,
especially spiky noise
Grafana is not the right tool for doing such pre-processing, so depending on
what you are currently using to collect this data and put it into a data
store, it might be better looking at extending that to store, as I suggested,
the raw data in one database (or at least, one table, if you use a table-
oriented store), and the processed data in another - then you can re-run the
processing any time you like with different parameters until you like the
output it generates.
Hi, Tanks you for all theses information. Very helpful. I understand that the best solution is to pre-process my data. The reason I’m trying to process data “on the fly” is only to reduce storage. And because I’m newbie I hoped that something that’s very popular already exist for that kind of application. Anyway, thanks a lot. Best regards.
