Decrypting Docker Container Names in Grafana with Loki and Promtail

Grafana Configuration
, ,
1

Hello, everyone!

I am Junior DevOps Engineer. My task is to configure Docker log monitoring for around 70 containers. Consolidating all logs into a single panel is not ideal, especially during stress tests and monitoring Docker applications. I want to be able to select a Docker container application in a panel and see the corresponding logs. I have achieved this, but the problem is that the container names are encrypted, and I need to decrypt them.

My question is: How can I decrypt the container names using the Loki data source and Promtail?

Additionally, I noticed that the Prometheus data source displays decrypted container names, but the Loki data source does not. Here are my configurations for both:

Also, I am running Loki and Promtail like normal application with configured systemd daemon.
Loki:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096


ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
  - from: 2021-03-08
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  boltdb:
    directory: /tmp/loki/index

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Promtail:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: cadvisor
  static_configs:
  - targets: ['localhost:8080/containers/']
    labels:
      job: cadvisonr
      __path__: /containers
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - regex:
      expression: (?P<container_name>(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      # tag:
      stream:
      container_name:
  - output:
      source: output

Also, I know about loki-driver-docker plugin, but I tried to run it and nothing changes for me, and documentation is terrible, it says just run it and almost nothing about configuration or how to check if it is working correct.

So, I would be glad to hear any ideas and suggestions. Please, don’t send me 58572 forum topic and github/gist pages with labels config. I tried all of it and it didn’t work for me. Forum is the last chance for me to solve this problem.

UPD: Currently, it looks like this:

image
image1091×136 43.5 KB

And I want it to be decrypted like in cadvisor:
image