Extract values from log message

Grafana Elasticsearch
1

Hi
I am using Grafana 7.0.3 with ElasticSearch 7.4
I have logs in my system with various values. I want to be able to extract those values from the log field, and use them to create the graph (specifically i have an organization (enum) value, and time value in ms - both part of the log message)
I cant find how to get these values from the log string using Lucene Queries.

Is this even possible ? and If so were can I find the syntax.

Thanks.

2

Very possible, use logstash…
The script depends on the log format that you want to parse.

For the specific log format, you need to build the specific grok syntax.

Regards,
Fadjar Tandabawana

3

Could you please give an example of this?
I am working with Grafana, Lucene, Elasticsearch right now and trying to find a way to do this.

4

For start, you can visit this URL:

Then if you want to have complex log content, you can learn about grok pattern.
Every pattern have their own script, so for example, you need to know the content of the logs.

Regards,
Fadjar

5

Thanks for the reply!
Let me ask you cause I am not sure if I can do this in Grafana…
I need to get a series of logs showing a failure then extract the id values in the message body of those logs and THEN identify successful logs containing the same ID.

tl;dr
I need to extract id value from the message body of logs then do another query searching for “successful” AND {id_value}

Thanks

6

Hi…

Maybe I misinterpreted your message in the first… Sorry, my wrong…

I assume that you have log that not yet pushed to Elasticsearch. In this case you need logstash or filebeat to “convert” your log to Elasticsearch.

If your log already in the elasticsearch, it’s simple enough to show in Grafana, just build the query like you build in Kibana, then use it in Grafana.

If you want to get the id_value and successful you just need to use grok pattern in the Logstash. Dont’t forget to extract timestamp for the presentation in Grafana.

If you have sample of the log, I can help to build the grok pattern. You can message me in private if there some sensitive information about the log…

Regards,
Fadjar

7

Oh thank you so much!