Promtail not parsing logs

[stefnats]

Hey there,

I’m struggling to set up promtail correctly.

I am using a Kubernetes cluster with the kube-prometheus-stack and i also installed grafana/loki-stack.
Promtail works, Loki works and i see everything in the explore tab.
Now, my struggle is to parse the output.

The output is generated by my app to stdout using Log4j2. This is an example message:

{"instant":{"epochSecond":1675415288,"nanoOfSecond":596145731},"thread":"pool-4-thread-3","level":"INFO","loggerName":"com.myapp.SomeClass","message":"This is some log line","contextMap":{"customerID":"3736","galleryId":"4357"},"endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":62,"threadPriority":5}

This is my config:

server:
  log_level: info
  http_listen_port: 3101


clients:
  - url: http://loki:3100/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
  - job_name: myapp-logs
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: 
            - myapp-ns
    pipeline_stages:
      - cri: { }
      - json:
          expressions:
            timestamp: instant_epochSecond
            level: level
            thread: thread
            class: loggerName
            message: message
            context: contextMap
            customerID: contextMap_customerID
            galleryId: contextMap_galleryId
      - labels:
          level:
          class:
          context:
          thread:
          customerID: 
          galleryId: 
      - timestamp:
          format: Unix
          source: timestamp
      - output:
          source: message
  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: { }
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_instance
          - __meta_kubernetes_pod_label_release
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: instance
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: node_name
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - namespace
          - app
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__
      - action: replace
        regex: true/(.*)
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
          - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
          - __meta_kubernetes_pod_container_name
        target_label: __path__



limits_config:

Questions:

1. How can i configure Promtail so, that the message is parsed and can be read directly in the explore tab of Grafana?
2. How do i add labels (or even static labels)?

Thanks in advance!

[b0b]

It is recommended to keep the number of labels as low as possible

Labels | Grafana Loki documentation

Don’t try to use Loki like a fully indexed JSON store like e.g. Elasticsearch.

You parse the logs at query time
Log queries | Grafana Loki documentation (check the JSON section)

You then select which field(s) you want to view in Grafana with (+)

[stefnats]

Hey b0b,

thanks so much for your answer!
You are right, i will try and ditch most labels.
However i need to know the job (aka process or pod name) and i need to know the customer id. There are a couple hundred customers.

Would you recommend another log output format besides JSON or what do you think prevents the config from working what i want to do?

Currently, i don’t really get a grasp and a beginning on how to solve that, maybe you can give me a few tips on how to write the correct config?

Thanks!

[b0b]

Hi @stefnats ,

I setup a Loki POC some time ago and have not gotten back to it to complete the project yet so not sure how great my answers will be…

Here is my scrape_config, maybe there is something useful there

      - name: kubernetes_pods
        positions:
          filename: /tmp/positions_pods.yaml
        scrape_configs:
          - job_name: kubernetes_pods
            kubernetes_sd_configs:
              - role: pod
            pipeline_stages:
              - docker: {}
            relabel_configs:
              - source_labels:
                - __meta_kubernetes_pod_controller_name
                target_label: __service__
              - source_labels:
                - __meta_kubernetes_pod_node_name
                target_label: __host__
              - action: labelmap
                regex: __meta_kubernetes_pod_label_(app|project|service)
              - action: replace
                replacement: $1
                source_labels:
                - name
                target_label: job
              - action: replace
                source_labels:
                - __meta_kubernetes_namespace
                target_label: namespace
              - action: replace
                source_labels:
                - __meta_kubernetes_pod_name
                target_label: pod
              - action: replace
                source_labels:
                - __meta_kubernetes_pod_container_name
                target_label: container
              - replacement: /var/log/pods/*$1/*.log
                separator: /
                source_labels:
                - __meta_kubernetes_pod_uid
                - __meta_kubernetes_pod_container_name
                target_label: __path__

I’m really only using Kubernetes resource labels so I have no personal experience from trying to capture something for the log messages.

Comparing our configs, you probably want to change your cri pipeline stage to docker. That will probably already get you close to what you want to do. You should be able to pass the log field to the JSON parser to extract the field you want as a label.