Loki Custom Config File in Podman

[dduncan0622]

When installing loki in podman I am not sure how to confirm if the loki-config.yaml I have on my system is being used in the container. I have run the setup as it is mentioned in the grafana installation using this command:

podman run --name loki -d -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 -config.file=/mnt/config/loki-config.yaml

In rootless podman this command does not work but it will in rootfull. When I say that it works in rootfull I do not have any evidence that it actually used my custom config file.

I have one issue in my test environment which many posts have mentioned to add the following line to the config file: frontend:
max_outstanding_per_tenant: 2048

Either this fix does not work because the issue is still there or it is not reading my config file and still reading the default file when installing loki.

I have also tried the steps as outlined here: GitHub - prosenjitjoy/Grafana-Loki-and-Promtail-with-Podman: Grafana Loki and Promtail setup guide using podman and this will run loki but again with no real proof that it is in fact using the custom config file. Any help is greatly appreciated.

[tonyswumac]

I’ve not tried to run Loki on podman, but there is a /config API endpoint you can use to get the running configuration from a Loki container.

[yosiasz]

Welcome

This is what I have in windows environment, but I have not touched it in a while and flog section is just to mock data to consume

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.8.0
    ports:
      - "3101:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yml:/etc/loki/loki-config.yml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.8.0
    volumes:
      - promtail:/var/log
      - ./promtail-local-config.yaml:/etc/promtail/promtail-config.yaml
    command: -config.file=/etc/promtail/promtail-config.yaml
    networks:
      - loki

  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy 
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3953:3000"
    networks:
      - loki

  flog:
    image: mingrammer/flog
    command: -f json -d 1s -l
    networks:
      - loki

volumes:
  promtail:
  sock:

[dduncan0622]

is your instance running the local-config.yaml or the loki-config.yaml?

[yosiasz]

Mine is running the following
local-config.yml

Also I would highly recommend installing podman desktop until you fully feel comfortable navigating in the command line.

[dduncan0622]

We are on RHEL 8 so everything is command line Podman is pretty much replacing docker especially when running RHEL .

[yosiasz]

So did the compose approach work?

[dduncan0622]

I am not sure. The issue didn’t go away and I have no way to verify that Loki is indeed using my config file. I tried to check the log but it doesn’t appear to provide information that it is. We are not using docker compose as we are configuring in podman based on install directions supplied by Grafana. Those do not seem to work unless you are in rootfull mode which defeats the purpose of using podman. Podman is new to us and we are just trying to get it all figured out.

[yosiasz]

You have to test it end to end. By that I mean you have to look in grafana if logs are getting pulled in

[dduncan0622]

Loki works and we can get logs. The issue we ran into is that all of the sudden the data in the dashboard goes away and we are left with the error : to many outstanding requests
The loki config file by default is not large but there are attributes that can be added apparently to build it out however you need it. One fix mentioned for this is to Increase max_outstanding_per_tenant to 2048. This was mentioned in https://github.com/grafana/loki/issues/4613

I have seen others report this issue with not much response. I am trying to see if this works for me but I am looking to find a way to show proof that Loki really is using the config file I give it and not its default config file. I suppose one way would be to create a volume that points to the directory loki stores the config file in. Once you run the container you would then see the config file in the volume. If you stopped the container then in theory you could change the config file and then start the container again. Down side to this option is that you need root access to do this. I also feel this may not be standard practice.

[yosiasz]

@dduncan0622

file I give it and not its default config file.

Is this documented that it uses its own config? Best way to test that is run it without a config

[dduncan0622]

you can run it without giving the config and it will work I have done this already. Just in trying to learn I have run the commands multiple ways.

If you run podman volume create loki_confg you get your volume

run → podman run --name loki -d -v loki_config:/etc/loki -p 3100:3100 grafana/loki:latest

This will create a container and then you can go to the directory of the volume you created and see the config file loki is using.

[yosiasz]

can you in your config file add something unique that can tell you that it is using your config?