Promtail: memory consumption, possible leak?

Hi everyone!

We are using the Promtail Helm Chart (Chart version 6.7.4, with Promtail 2.7.0) on EKS to send all Pod/Docker logs to Loki. The Promtail Pods on the EKS nodes show a slow but steady increase in memory usage over time, e.g.:

IMHO this looks very typical for a memory leak. Not every EKS node gets old enough for this to become a real problem, but some nodes do and they experience OOM kills for Promtail. However, I wanted to make sure if this is expected/unexpected behavior before opening an issue on GitHub - maybe I’m missing some configuration option?

We are using the default values of the Helm Chart (with exception of the Loki URL and some podAnnotations and tolerations).

I found promtail memory leak · Issue #6968 · grafana/loki · GitHub and which let’s you limit max_streams, but I’m not sure how to determine how many streams I need or even how many I’m currently using.

Any hints or tips?

It really frustrates me that the Grafana Loki/Promtail team doesn’t respond to well-presented, likely important issues like this. I’ve never gotten a response on this forum, even when I found an error in an official tutorial of theirs and wanted to know how to fix it.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.