Getting this error:
[2022/06/21 21:17:10] [error] [output:loki:loki.0] loki-stack:3100, HTTP status=400 Not retrying.
entry with timestamp 2022-06-21 20:32:01.696331203 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="f787b5f7d", k8s_app="kube-proxy", pod_template_generation="6", updateTime="1653791485"},
total ignored: 1 out of 1
Running EKS v 1.22
Installed Loki using helm along with the following values:
loki:
enabled: true
persistence:
enable: true
size: 50Gi
storageClassName: efs
configured the fluent bit to collect all the logs. The configuration is the following:
using the following configuration:
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 64M
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5M
storage.type filesystem
Skip_Long_Lines On
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Mem_Buf_Limit 512M
storage.type filesystem
Read_From_Tail On
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
name loki
match *
host loki-stack
remove_keys kubernetes,stream
auto_kubernetes_labels on
storage.total_limit_size 512M
log_level {{ .Values.logLevel }}