Loki driver on docker and getting stuck container

Grafana Loki
1

Hello everyone.
Faced the following situation.
I have a container in docker, sending logs through the Loki driver to the Loki server. Here is a snippet from docker-compose:

...
    ports:
      - '5000'
    restart: always
    logging:
      driver: loki
      options:
        loki-url: http://my_grafana_loki_server:3100/loki/api/v1/push
...

Everything works great. But if for some reason the Loki server is unavailable (in my case, for example, the firewall prohibits access from the IP address to the port or just the server is not available), the container becomes unmanageable and sometimes completely malfunctioning. Stopping the container in this state takes a noticeable time up to 10 minutes.
Perhaps there are some practices to avoid this situation if the server with Loki is temporarily not working or Loki port is blocked, etc…
Thanks.

2

Hi,

I’m sorry you hit that. It’s a known issue. See Docker driver | Grafana Labs and if loki is not reachable and loki-docker-driver is activated, containers apps stops and cannot be stopped/killed · Issue #2361 · grafana/loki · GitHub

You can either scrape the log files or up vote A Docker service discovery to Promtail · Issue #4703 · grafana/loki · GitHub

3

Hi
Thanks a lot!
This is my working configuration:

   logging:
      driver: loki
      options:
        loki-url: http://my_grafana_loki_server:3100/loki/api/v1/push
        mode: non-blocking
        max-buffer-size: 4m
        loki-retries: "3"
4

Thank you, you saved us big time

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.