Hi,
I’ve setup a Loki instance in a Kubernetes cluster, based on this chart : helm-charts/charts/loki-simple-scalable at main · grafana/helm-charts · GitHub
Logs are pushed by some Fluentd pods, then we consume them with Grafana for dashboarding and alerting purpose. We also use an Azure Storage account for persistence, instead of having logs stored directly in some volumes of this cluster.
The issue I’m facing is that some logs seem to be missing. At some times, the system can be under high traffic, with a good amount of logs being pushed continuously. But when querying (through Grafana, or using a LogCLI pod), some logs don’t show up at all. As if it was lost during the write process into Loki.
In Loki container logs, I see no particular error or warning. I don’t really know where to look to solve this issue. Do you have any hint on why it could happen ?
Thanks in advance