Promtail to promtail authentication (oauth2 through external idp)

Grafana Loki
, , , ,
1

Hello,

I am trying to configure a secure architecture where each of my ~ 10 servers can send their logs to my central monitoring architecture.

From what I understood, so far my best option is leveraging the promtail --> promtail authentication to have something like this:

Server_{1..n}(promtail) --> Server_monitoring(promtail --> loki --> grafana);

Even more precisely, everything running on Server_monitoring is authenticated through OAuth2 (using gitlab.com as an identity provider), configure with caddy:

Server_{1..n}(promtail) --> Server_monitoring( caddy+gitlab_oauth2 --> (promtail --> loki --> grafana)) ;

I found the clients: documentation Configuration | Grafana Loki documentation and tried to adapt it:

clients:
  - url: https://protmail.server_monitoring.domain/loki/api/v1/push
    oauth2:
      client_id:     'client_id_obfuscated'
      client_secret: 'client_secret_obfuscated'
      #scopes: "openid"
      token_url: https://gitlab.com/oauth/token

… but it doesn’t propagate anything, nor does it prints any error in the logs:

My full (oauth Id obfuscated) configuration is the following for the client (server_n) promtail:

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https://protmail.server_monitoring.domain/loki/api/v1/push
    oauth2:
      client_id:     'client_id_obfuscated'
      client_secret: 'client_secret_obfuscated'
      #scopes: "openid"
      token_url: https://gitlab.com/oauth/token

scrape_configs:

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        log: log
        stream: stream
        time: time
        tag: attrs.tag
        compose_project: attrs."com.docker.compose.project"
        compose_service: attrs."com.docker.compose.service"
        stack_name: attrs."com.docker.stack.namespace"
        swarm_service_name: attrs."com.docker.swarm.service.name"
        swarm_task_name: attrs."com.docker.swarm.task.name"
  - regex:
      expression: "^/var/lib/docker/containers/(?P<container_id>.{12}).+/.+-json.log$"
      source: filename
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      stream:
      container_id:
      tag:
      compose_project:
      compose_service:
      stack_name:
      swarm_service_name:
      swarm_task_name:
  - output:
      source: log

and I’m launching promtail through docker-compose.yml:

services:
  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    #<<: *logging
    volumes:
      - /var/log:/var/log
      - ./promtail:/etc/promtail/
    command: -config.file=/etc/promtail/promtail-config.yaml  -print-config-stderr
    restart: always

The only logs I get are the following:

$ sudo docker-compose -f docker-compose.yml --env-file env.ud1 up --no-deps -d --force-recreate promtail && make logsf-promtail
[+] Running 1/1
 ⠿ Container promtail  Started                                                             1.7s
sudo docker-compose -f docker-compose.yml --env-file env.ud1 logs --tail=500 -tf promtail
promtail  | 2022-12-27T21:47:48.850524030Z ---
promtail  | 2022-12-27T21:47:48.850553946Z # Loki Config
promtail  | 2022-12-27T21:47:48.850562242Z # (version=2.7.1, branch=HEAD, revision=e0af1cc8a)
promtail  | 2022-12-27T21:47:48.850568784Z server:
promtail  | 2022-12-27T21:47:48.850580065Z   http_listen_network: tcp
promtail  | 2022-12-27T21:47:48.850584313Z   http_listen_address: 0.0.0.0
promtail  | 2022-12-27T21:47:48.850588240Z   http_listen_port: 9080
promtail  | 2022-12-27T21:47:48.850591847Z   http_listen_conn_limit: 0
promtail  | 2022-12-27T21:47:48.850600744Z   grpc_listen_network: tcp
promtail  | 2022-12-27T21:47:48.850604832Z   grpc_listen_address: ""
promtail  | 2022-12-27T21:47:48.850608669Z   grpc_listen_port: 9095
promtail  | 2022-12-27T21:47:48.850612586Z   grpc_listen_conn_limit: 0
promtail  | 2022-12-27T21:47:48.850620521Z   tls_cipher_suites: ""
promtail  | 2022-12-27T21:47:48.850624839Z   tls_min_version: ""
promtail  | 2022-12-27T21:47:48.850628496Z   http_tls_config:
promtail  | 2022-12-27T21:47:48.850635449Z     cert_file: ""
promtail  | 2022-12-27T21:47:48.850639076Z     key_file: ""
promtail  | 2022-12-27T21:47:48.850643063Z     client_auth_type: ""
promtail  | 2022-12-27T21:47:48.850646510Z     client_ca_file: ""
promtail  | 2022-12-27T21:47:48.850653593Z   grpc_tls_config:
promtail  | 2022-12-27T21:47:48.850657721Z     cert_file: ""
promtail  | 2022-12-27T21:47:48.850661288Z     key_file: ""
promtail  | 2022-12-27T21:47:48.850664864Z     client_auth_type: ""
promtail  | 2022-12-27T21:47:48.850672379Z     client_ca_file: ""
promtail  | 2022-12-27T21:47:48.850676136Z   register_instrumentation: true
promtail  | 2022-12-27T21:47:48.850679642Z   graceful_shutdown_timeout: 30s
promtail  | 2022-12-27T21:47:48.850683299Z   http_server_read_timeout: 30s
promtail  | 2022-12-27T21:47:48.850690893Z   http_server_write_timeout: 30s
promtail  | 2022-12-27T21:47:48.850694901Z   http_server_idle_timeout: 2m0s
promtail  | 2022-12-27T21:47:48.850699860Z   grpc_server_max_recv_msg_size: 4194304
promtail  | 2022-12-27T21:47:48.850707144Z   grpc_server_max_send_msg_size: 4194304
promtail  | 2022-12-27T21:47:48.850711151Z   grpc_server_max_concurrent_streams: 100
promtail  | 2022-12-27T21:47:48.850714678Z   grpc_server_max_connection_idle: 2562047h47m16.854775807s
promtail  | 2022-12-27T21:47:48.850718886Z   grpc_server_max_connection_age: 2562047h47m16.854775807s
promtail  | 2022-12-27T21:47:48.850725949Z   grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
promtail  | 2022-12-27T21:47:48.850729626Z   grpc_server_keepalive_time: 2h0m0s
promtail  | 2022-12-27T21:47:48.850733674Z   grpc_server_keepalive_timeout: 20s
promtail  | 2022-12-27T21:47:48.850737892Z   grpc_server_min_time_between_pings: 5m0s
promtail  | 2022-12-27T21:47:48.850744694Z   grpc_server_ping_without_stream_allowed: false
promtail  | 2022-12-27T21:47:48.850748962Z   log_format: logfmt
promtail  | 2022-12-27T21:47:48.850752349Z   log_level: info
promtail  | 2022-12-27T21:47:48.850759923Z   log_source_ips_enabled: false
promtail  | 2022-12-27T21:47:48.850764111Z   log_source_ips_header: ""
promtail  | 2022-12-27T21:47:48.850767878Z   log_source_ips_regex: ""
promtail  | 2022-12-27T21:47:48.850772076Z   log_request_at_info_level_enabled: false
promtail  | 2022-12-27T21:47:48.850779249Z   http_path_prefix: ""
promtail  | 2022-12-27T21:47:48.850783107Z   external_url: ""
promtail  | 2022-12-27T21:47:48.850786984Z   health_check_target: null
promtail  | 2022-12-27T21:47:48.850791222Z   disable: false
promtail  | 2022-12-27T21:47:48.850802453Z   enable_runtime_reload: false
promtail  | 2022-12-27T21:47:48.850806992Z client:
promtail  | 2022-12-27T21:47:48.850810568Z   url: ""
promtail  | 2022-12-27T21:47:48.850813925Z   batchwait: 1s
promtail  | 2022-12-27T21:47:48.850821539Z   batchsize: 1048576
promtail  | 2022-12-27T21:47:48.850825597Z   follow_redirects: false
promtail  | 2022-12-27T21:47:48.850837379Z   enable_http2: false
promtail  | 2022-12-27T21:47:48.850844833Z   backoff_config:
promtail  | 2022-12-27T21:47:48.850848620Z     min_period: 500ms
promtail  | 2022-12-27T21:47:48.850852156Z     max_period: 5m0s
promtail  | 2022-12-27T21:47:48.850856034Z     max_retries: 10
promtail  | 2022-12-27T21:47:48.850863518Z   timeout: 10s
promtail  | 2022-12-27T21:47:48.850867786Z   tenant_id: ""
promtail  | 2022-12-27T21:47:48.850871613Z   stream_lag_labels: ""
promtail  | 2022-12-27T21:47:48.850875360Z clients:
promtail  | 2022-12-27T21:47:48.850883105Z - url: https://promtail.server_monitoring.domain/loki/api/v1/push
promtail  | 2022-12-27T21:47:48.850887072Z   batchwait: 1s
promtail  | 2022-12-27T21:47:48.850890579Z   batchsize: 1048576
promtail  | 2022-12-27T21:47:48.850894296Z   oauth2:
promtail  | 2022-12-27T21:47:48.850901379Z     client_id: <obfuscated>
promtail  | 2022-12-27T21:47:48.850905467Z     client_secret: <secret>
promtail  | 2022-12-27T21:47:48.850909805Z     client_secret_file: ""
promtail  | 2022-12-27T21:47:48.850917239Z     token_url: https://gitlab.com/oauth/token
promtail  | 2022-12-27T21:47:48.850921186Z   follow_redirects: false
promtail  | 2022-12-27T21:47:48.850925955Z   enable_http2: false
promtail  | 2022-12-27T21:47:48.850929752Z   backoff_config:
promtail  | 2022-12-27T21:47:48.850936976Z     min_period: 500ms
promtail  | 2022-12-27T21:47:48.850940803Z     max_period: 5m0s
promtail  | 2022-12-27T21:47:48.850944770Z     max_retries: 10
promtail  | 2022-12-27T21:47:48.850948758Z   timeout: 10s
promtail  | 2022-12-27T21:47:48.850955791Z   tenant_id: ""
promtail  | 2022-12-27T21:47:48.850959618Z   stream_lag_labels: ""
promtail  | 2022-12-27T21:47:48.850963716Z positions:
promtail  | 2022-12-27T21:47:48.850967704Z   sync_period: 10s
promtail  | 2022-12-27T21:47:48.850975148Z   filename: /tmp/positions.yaml
promtail  | 2022-12-27T21:47:48.850979135Z   ignore_invalid_yaml: false
promtail  | 2022-12-27T21:47:48.850982952Z scrape_configs:
promtail  | 2022-12-27T21:47:48.850989765Z - job_name: containers
promtail  | 2022-12-27T21:47:48.850993462Z   pipeline_stages:
promtail  | 2022-12-27T21:47:48.850996888Z   - json:
promtail  | 2022-12-27T21:47:48.851000946Z       expressions:
promtail  | 2022-12-27T21:47:48.851007949Z         compose_project: attrs."com.docker.compose.project"
promtail  | 2022-12-27T21:47:48.851012157Z         compose_service: attrs."com.docker.compose.service"
promtail  | 2022-12-27T21:47:48.851015884Z         log: log
promtail  | 2022-12-27T21:47:48.851019791Z         stack_name: attrs."com.docker.stack.namespace"
promtail  | 2022-12-27T21:47:48.851027686Z         stream: stream
promtail  | 2022-12-27T21:47:48.851031393Z         swarm_service_name: attrs."com.docker.swarm.service.name"
promtail  | 2022-12-27T21:47:48.851035180Z         swarm_task_name: attrs."com.docker.swarm.task.name"
promtail  | 2022-12-27T21:47:48.851042574Z         tag: attrs.tag
promtail  | 2022-12-27T21:47:48.851046261Z         time: time
promtail  | 2022-12-27T21:47:48.851049928Z   - regex:
promtail  | 2022-12-27T21:47:48.851053585Z       expression: ^/var/lib/docker/containers/(?P<container_id>.{12}).+/.+-json.log$
promtail  | 2022-12-27T21:47:48.851061600Z       source: filename
promtail  | 2022-12-27T21:47:48.851065177Z   - timestamp:
promtail  | 2022-12-27T21:47:48.851069194Z       format: RFC3339Nano
promtail  | 2022-12-27T21:47:48.851073172Z       source: time
promtail  | 2022-12-27T21:47:48.851081568Z   - labels:
promtail  | 2022-12-27T21:47:48.851085435Z       compose_project: null
promtail  | 2022-12-27T21:47:48.851089352Z       compose_service: null
promtail  | 2022-12-27T21:47:48.851092869Z       container_id: null
promtail  | 2022-12-27T21:47:48.851099782Z       stack_name: null
promtail  | 2022-12-27T21:47:48.851103849Z       stream: null
promtail  | 2022-12-27T21:47:48.851107226Z       swarm_service_name: null
promtail  | 2022-12-27T21:47:48.851114379Z       swarm_task_name: null
promtail  | 2022-12-27T21:47:48.851118277Z       tag: null
promtail  | 2022-12-27T21:47:48.851121623Z   - output:
promtail  | 2022-12-27T21:47:48.851125280Z       source: log
promtail  | 2022-12-27T21:47:48.851132644Z   static_configs:
promtail  | 2022-12-27T21:47:48.851136210Z   - targets:
promtail  | 2022-12-27T21:47:48.851139667Z     - localhost
promtail  | 2022-12-27T21:47:48.851143414Z     labels:
promtail  | 2022-12-27T21:47:48.851151329Z       __path__: /var/lib/docker/containers/*/*log
promtail  | 2022-12-27T21:47:48.851155026Z       job: containerlogs
promtail  | 2022-12-27T21:47:48.851158793Z target_config:
promtail  | 2022-12-27T21:47:48.851162179Z   sync_period: 10s
promtail  | 2022-12-27T21:47:48.851169313Z   stdin: false
promtail  | 2022-12-27T21:47:48.851173180Z limits_config:
promtail  | 2022-12-27T21:47:48.851176767Z   readline_rate: 10000
promtail  | 2022-12-27T21:47:48.851184221Z   readline_burst: 10000
promtail  | 2022-12-27T21:47:48.851187978Z   readline_rate_drop: true
promtail  | 2022-12-27T21:47:48.851191785Z   max_streams: 0
promtail  | 2022-12-27T21:47:48.851195802Z tracing:
promtail  | 2022-12-27T21:47:48.851203527Z   enabled: true
promtail  | 2022-12-27T21:47:48.851207444Z
promtail  | 2022-12-27T21:47:48.851211041Z
promtail  | 2022-12-27T21:47:48.949321575Z level=info ts=2022-12-27T21:47:48.85189134Z caller=promtail.go:123 msg="Reloading configuration file" md5sum=99229b92ee1e3994a6aad5511810b9a9
promtail  | 2022-12-27T21:47:48.949354296Z level=info ts=2022-12-27T21:47:48.853190261Z caller=server.go:323 http=[::]:9080 grpc=[::]:9095 msg="server listening on addresses"
promtail  | 2022-12-27T21:47:48.949359406Z level=info ts=2022-12-27T21:47:48.853319033Z caller=main.go:171 msg="Starting Promtail" version="(version=2.7.1, branch=HEAD, revision=e0af1cc8a)"
promtail  | 2022-12-27T21:47:48.949362231Z level=warn ts=2022-12-27T21:47:48.853335273Z caller=promtail.go:220 msg="enable watchConfig"
promtail  | 2022-12-27T21:47:53.949498530Z level=info ts=2022-12-27T21:47:53.853343061Z caller=filetargetmanager.go:352 msg="Adding target" key="/var/lib/docker/containers/*/*log:{job=\"containerlogs\"}"

Any suggestion on how to make this work would be greatly appreciated !

2

in short : how do you configure oauth2 so that promtail sends the client_id and client_secrets to the webserver that checks it before granting access to loki ?

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.