Struggle to add label to logs on loki deployed with helm chart

Grafana Loki
,
1

Hi,
I deployed Loki and Promtail using the grafana Helm Chart and I struggle to configure it.

As a simple configuration, I would like to add a specific label (an UUID). To do so, I use the specific yaml :

config:
  lokiAddress: http://loki-headless.admin:3100/loki/api/v1/push
  snippets:
      extraScrapeConfigs: |
        - job_name: dashboard
          kubernetes_sd_configs:
            - role: pod
          pipeline_stages:
            - docker: {}
            - match:
                selector: '{container = "dashboard"}'
                stages:
                  - regex:
                      expression: '(?P<loguuid>[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12})'
                  - labels:
                      loguuid:

Which is deployed with the command:
helm upgrade -install promtail -n admin grafana/promtail -f promtail.yaml

Of course, I still don’t have the label in grafana.
Can someone tell me what I did wrong?

2

Hi @guillaumegillet ,

I don’t use helm and actually I do not use Promtail, but grafana-agent should use the same config.

This is how my grafana-agent job config looks like. Maybe it helps although it is not a direct comparison to your situation.

        positions:
          filename: /tmp/positions_pods.yaml
        scrape_configs:
          - job_name: kubernetes_pods
            kubernetes_sd_configs:
              - role: pod
            pipeline_stages:
              - docker: {}
            relabel_configs:
              - source_labels:
                - __meta_kubernetes_pod_controller_name
                target_label: __service__
              - source_labels:
                - __meta_kubernetes_pod_node_name
                target_label: __host__
              - action: labelmap
                regex: __meta_kubernetes_pod_label_(app|project|service)
              - action: replace
                replacement: $1
                source_labels:
                - name
                target_label: job
              - action: replace
                source_labels:
                - __meta_kubernetes_namespace
                target_label: namespace
              - action: replace
                source_labels:
                - __meta_kubernetes_pod_name
                target_label: pod
              - action: replace
                source_labels:
                - __meta_kubernetes_pod_container_name
                target_label: container
              - replacement: /var/log/pods/*$1/*.log
                separator: /
                source_labels:
                - __meta_kubernetes_pod_uid
                - __meta_kubernetes_pod_container_name
                target_label: __path__
3

Thx @b0b,

Unfortunately I am looking to extract something inside the log body. Unless I am wrong, your config only fetch info from kubernetes api

4

Sorry. i don’t know what I thought I was reading… In my head you where trying to add something that was available as Kubernetes metadata.

5

You are the only person I found having the same issues I was going through, so I suppose others will benefit of the solution:

Thanks to this post on stackoverflow:

https://stackoverflow.com/questions/63414638/promtail-ignores-extrascrapeconfigs

I was able to implement the correct solution for the exact same issue you are having.

Apparently the fact that extraScrapeConfigs is appended to the end of the configuration affects promtail ability to add custom labels. So, instead of using extraScrapeConfigs, use the pipelineStages. On loki-stack 2.2.0, this the configuration that worked for me:

promtail:
  enabled: true
  pipelineStages:
    - cri: {}
    - regex:
        expression: '^.*[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} (?P<level>[A-Z]+)  \[\[(?P<sourcefile>.*)\]\].*'

    - labels:
        level:
        sourcefile:

I hope it helps

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.