Extract Metrics using Loki Docker Driver and Container

Grafana Loki
1

Loki Docker Driver plug-in: grafana/loki-docker-driver:2.1.0
Loki Docker image: grafana/loki:2.1.0

I’m using Loki to parse JSON formatted log entries from Docker containers with the Loki Docker Driver plugin. My goal is to create labels that I can use in Loki, as well as produce metrics that I can export to Prometheus.

A sample log entry is as follows:

{"L":"INFO","T":"2021-01-30T06:17:18.876Z","N":"piecestore","M":"download started","Piece ID":"6CYNNMUS6DVVCXOE4N7JNFKXETIZVYBX4PSAVKPY3WPN7VKLVUPQ","Satellite ID":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","Action":"GET_REPAIR"}

The docker-compose.yml file includes the following pipeline stages for the logging: section:

x-logger: &logger
  logging:
    driver: loki
    options:
      loki-url: "http://172.18.0.250:3100/loki/api/v1/push"  # Static IP for Loki container on user-defined bridge
      loki-external-labels: job=storj
      loki-pipeline-stages: |
        - json:
            expressions:
              loglevel: L
              timestamp: T
              message: M
              satellite: '"Satellite ID"'
              action: Action
              error: error
              errorVerbose: errorVerbose
              available_space: '"Available Space"'
              size: Size
        - labels:
            loglevel:
            satellite:
            action:
            error:
        - metrics:
            download_started:
              type: counter
              description: "Total number of downloads started"
              prefix: storj_pieces_
              max_idle_duration: 24h
              source: message
              config:
                value: "download started"
                action: inc

Since there isn’t a Promtail container in this setup, I would expect that the metrics would populate at the Loki container which is accessible on http://localhost:3100/metrics. However, I see no metrics beginning with the storj_pieces_ prefix as specified above. I can confirm that the JSON parsing and label assignment is successful, having queried the logs using LogQL in Grafana.

Is there a step I have missed in order to get metrics to be available at the Loki container target?

2

I’m learning now that Loki cannot expose metrics directly via the Docker driver and so it first needs to be pushed to Promtail.

3

I have solved this by using the loki_push_api in Promtail to first send the logs from the Docker driver. I then run all my pipeline stages in Promtail, including exposing the metrics from Prometheus. Then the logs are pushed from Promtail to Loki.

4

Really appreciate you posting your follow-up and solution @fmoledina this is what I would have suggested for you as well! Thanks!

1 Like
5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.