Promtail JSON strings in FQDN format

Grafana Loki
,
1

I am configuring Promtail to scrape our Docker Logs and push them to Loki. It’s 90% of the way there, I just can’t add some labels.

Basically, we want an easy way to determine what Docker Compose Stack and Service the log is from. Docker allows us to expose some of its internal labels it generates in the log. This gives us the labels “com.docker.compose.project” and “com.docker.compose.service”. Adding these to our logging directive in docker gives us the following:

x-logging: &promtail
  driver: "json-file"
  options:
    tag: "{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}"
    labels: "com.docker.compose.project,com.docker.compose.service"

This works, I can see the labels being populated in our logs when I cat them directly. Here is an example:

{
  "log": "[Thu Mar 31 17:07:19 2022] [::ffff:127.0.0.1]:47450 Closing\n",
  "stream": "stderr",
  "attrs": {
    "com.docker.compose.project": "adminer",
    "com.docker.compose.service": "adminer",
    "tag": "adminer:4.8.1|r-dc-adminer-1|sha256:e7ff5f92e5090766b9aaf4df6c017b76b3dd7789e353c0fb8b3e65c70bdb8196|41489d34de20b37b3a2ef91f4284e5dad6a84e2822c1083fd9d3ce4a78db55cb"
  },
  "time": "2022-03-31T17:07:19.711881439Z"
}

I have tried every permutation I can think of to get this label on the Promtail side. Here’s what I think should work:

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https://<REDACTED>/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      host: ${DOCKER_HOST}
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        source: stream
        attrs:
  - json:
      expressions:
        tag:
        stack: "com.docker.compose.project"
        stack_service: "com.docker.compose.service"
      source: attrs
  - regex:
      expression: (?P<image_name>(?:[^|]*[^|])).(?P<container_name>(?:[^|]*[^|])).(?P<image_id>(?:[^|]*[^|])).(?P<container_id>(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      tag:
      source:
      image_name:
      container_name:
      image_id:
      container_id:
      stack:
      stack_service:
  - output:
      source: output

So far I have tried it with the periods replaced with underscores, with the “stack” and the “com.docker.compose.project” the other way around, with no renaming, just trying to get the field across, everything I can think of.

From a bit of playing around on the JMESPath website, is seems com.docker.compose.project will fail, but putting it in quotes should work.

Does anyone have any ideas or had success getting these labels into Promtail? I feel knowing the compose project is fairly basic in terms of data and such but it seems the fact they are FQDN based breaks everything.

2

I was just trying to get this working myself
The workaround for this is to use a replace stage to change com.docker.compose.project to something without dots
For example I used:

pipeline_stages:  
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - replace:
      expression: '(com.docker.compose.project)'
      source: attrs
      replace: 'docker_stack'
  - json:
      expressions:
        tag:
        docker_stack:
      source: attrs

This replaced com.docker.compose.project within the attrs string so that it could be pulled out as docker_stack

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.