Promtail: memory consumption, possible leak?

Grafana Loki
1

Hi everyone!

We are using the Promtail Helm Chart (Chart version 6.7.4, with Promtail 2.7.0) on EKS to send all Pod/Docker logs to Loki. The Promtail Pods on the EKS nodes show a slow but steady increase in memory usage over time, e.g.:

Auswahl_274
Auswahl_274745×281 23.7 KB

IMHO this looks very typical for a memory leak. Not every EKS node gets old enough for this to become a real problem, but some nodes do and they experience OOM kills for Promtail. However, I wanted to make sure if this is expected/unexpected behavior before opening an issue on GitHub - maybe I’m missing some configuration option?

We are using the default values of the Helm Chart (with exception of the Loki URL and some podAnnotations and tolerations).

I found promtail memory leak · Issue #6968 · grafana/loki · GitHub and https://github.com/grafana/loki/pull/7101 which let’s you limit max_streams, but I’m not sure how to determine how many streams I need or even how many I’m currently using.

Any hints or tips?

2

It really frustrates me that the Grafana Loki/Promtail team doesn’t respond to well-presented, likely important issues like this. I’ve never gotten a response on this forum, even when I found an error in an official tutorial of theirs and wanted to know how to fix it.

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.