When querying logs from containers and using logfmt, Loki NGinx gateway log messages are shown as erroneous, stating: Error: LogFmtParserErr. Loki is stated as a Compose application with Read, Write components separated and Gateway put in front (example deployment from GitHub).
All POST events from Loki container are shown with the warning sign in Grafana dashboard. Do I need to reformat the log for it to be accepted by logfmt?
If I restart container I do see some correct log messages, indicating that container have restarted, next to erroneous NGinx HTTP action log messages. So it looks like those log messages are not parsed correctly. This is how they do look in container json log file.
logfmt format is key=value pair separated by space, which is not what your nginx log looks like. For custom log format write your own pattern.
Since you are send all container logs, there will be some that won’t parse (for example your stdout and stderr logs are likely different. You can include __error__ = "" in your filter to remove the log lines that fail to parse.
How’s that NGinx are not key:value? Or logfmt explicitly expects log value to be in key:value format?
This is the log excerpt from the same container, running Loki Gateway component. So if I understand situation, I have to update Promtail config for job_name: docker and add additional match: stage to look for NGINX output and reformat that output to be in key:value. Something like:
You do not need to change your Promtail configuration to reformat your logs, nor do I recommend you to. You can write your own pattern to parse the logs after they land in Loki.
Not sure I would agree. This is a log dashboard, which is given to end user. Therefore it needs to: 1. adhere to KISS 2. Be universal to show all logs.
In order to have that, I need my logs to be uniform. And for most part they really are: Windows and IIS, SystemD, rsyslog souces. It just the Loki Nginx container in the container stream stands out. So I rather fix this one in docker:{} pipeline stage, than make complex LogQL to gave exceptions in data stream handling.
HM… I have found another issue and I think this is why log’s are not parsed. I have updated pipeline to restructure logs and I still do get an error if the message has double quotes in it.
To test I have inserted a replacement string. If this test string contains double quotes, then logfmt fails to parse and shows only "Test string " as detected msg field. If string does not contain quotes, then log restructuring works and logfmt accepts log stream.
Question: how to quote in template pipeline?
Escaped single qote does not work and promtail fails to start:
'{{ if eq .tmp_msg "" }}{{ .Value }}{{ else }}level=info ts=2022-12-16T07:15:37.503910058Z caller=liauliau msg=\'{{ .tmp_msg }}\'{{ end }}'
I have updated NGinx logs to NOT contain double quotes (") and reformatted log messages using single quotes ('). This allows logfmt to properly parse container output with container pipeline having just this:
pipeline_stages:
- docker: {}
Did not expect that double quotes in string would mage string parser to stumble, however apparently this is how logfmt compatible logs should be structured.
Just tested with syslog and quoted string also breaks logfmt parser.
#!/bin/bash
for i in auth auth-priv cron daemon kern lpr mail mark news syslog user uucp local0 local1 local2 local3 local4 local5 local6 local7
do
for k in debug info notice warning err crit alert emerg
do
logger -p $i.$k "Test daemon message \"quoted string\" facility $i priority $k"
done
done
