My services are (mostly) outputting JSON, and I’m using the Grafana Fluent-Bit “grafana-loki” output of FluentBit. The “tail” input is correctly recognising JSON in the docker logs (I’m on k8s) and adding the “log_processed” tag. It’s configured like this:
filter-kubernetes.conf
:
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
parsers.conf
:
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
output-loki.conf
:
[OUTPUT]
Name grafana-loki
Match *
Url http://loki-writer:3100/api/prom/push
TenantID CQ-15396174
BatchWait 1
BatchSize 1048576
Labels {job="fluent-bit"}
RemoveKeys kubernetes,stream
AutoKubernetesLabels false
LabelMapPath /fluent-bit/etc/labelmap.json
LineFormat json
LogLevel warn
However, the result when read from Loki looks like this:
How can I make loki use the log fields as tags and use the timestamp from them?