Promtail does not increase positions, but set positions zero after a few hours. And then not report logs anymore

Grafana Loki
,
1

GitHub Issue: Promtail does not increase positions, but set positions zero after a few hours. And then not report logs anymore. · Issue #9864 · grafana/loki · GitHub

Environment

  • kubernetes 1.15, kubeadm, ubuntu 22.04
  • CONTAINER-RUNTIME: docker
  • loki 2.8.2, promtail 2.8.2
  • no log rotation

More information

  • loki-stack helm charts values.yaml
loki:
  image:
    registry: docker.io
    repository: grafana/loki
    tag: 2.8.2
    pullPolicy: IfNotPresent
  config:
    auth_enabled: false
    compactor:
      shared_store: filesystem
      working_directory: /data/loki/boltdb-shipper-compactor
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: inmemory
          replication_factor: 1
      wal:
        dir: /data/loki/wal
    limits_config:
      enforce_metric_name: false
      ingestion_burst_size_mb: 100
      ingestion_rate_mb: 50
      per_stream_rate_limit: 5MB
      per_stream_rate_limit_burst: 20MB
      reject_old_samples: true
      reject_old_samples_max_age: 168h
    schema_config:
      configs:
      - from: "2020-10-24"
        index:
          period: 24h
          prefix: index_
        object_store: filesystem
        schema: v11
        store: boltdb-shipper
    server:
      grpc_listen_port: 9095
      http_listen_port: 3100
    storage_config:
      boltdb_shipper:
        active_index_directory: /data/loki/boltdb-shipper-active
        cache_location: /data/loki/boltdb-shipper-cache
        cache_ttl: 24h
        shared_store: filesystem
      filesystem:
        directory: /data/loki/chunks
promtail:
  image:
    registry: docker.io
    repository: grafana/promtail
    tag: 2.8.2
    pullPolicy: IfNotPresent
  config:
    file: |
      server:
        http_listen_port: 3101
        log_level: info
      clients:
        - url: http://loki:3100/loki/api/v1/push
      positions:
        filename: /run/promtail/positions.yaml
      scrape_configs:
      - job_name: kubernetes-pod-label-app
        pipeline_stages:
          - docker: {}
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels:
          - __meta_kubernetes_pod_label_app
          regex: ''
          action: drop
        - regex: __meta_kubernetes_pod_label_(.+)
          action: labelmap
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_label_app
          separator: /
          target_label: job
        - source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          separator: /
          replacement: /var/log/pods/*$1/*.log
          target_label: __path__
  • docker daemon.json
{
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"]
}
  • promtail positions.yaml
positions:
  /var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log: "0"
  • log path
> ll /var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log

lrwxrwxrwx 1 root root 165 Jun 30 17:17 /var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log -> /var/lib/docker/containers/103f6f04fd7a429b7453dc54b901ae2d468cfcf6243a0df60073b8ac1d5b633c/103f6f04fd7a429b7453dc54b901ae2d468cfcf6243a0df60073b8ac1d5b633c-json.log

> ll -h /var/lib/docker/containers/103f6f04fd7a429b7453dc54b901ae2d468cfcf6243a0df60073b8ac1d5b633c/103f6f04fd7a429b7453dc54b901ae2d468cfcf6243a0df60073b8ac1d5b633c-json.log

-rw-r----- 1 root root 295M Jul  5 18:51 /var/lib/docker/containers/103f6f04fd7a429b7453dc54b901ae2d468cfcf6243a0df60073b8ac1d5b633c/103f6f04fd7a429b7453dc54b901ae2d468cfcf6243a0df60073b8ac1d5b633c-json.log
  • promtail pod logs:
level=info ts=2023-07-05T10:15:53.083081825Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log.20230705-181553 op=CREATE
level=info ts=2023-07-05T10:15:53.083185919Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log op=CREATE
level=info ts=2023-07-05T10:16:03.114822749Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log.20230705-181603 op=CREATE
level=info ts=2023-07-05T10:16:03.114948545Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log op=CREATE
level=info ts=2023-07-05T10:16:13.144539294Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log.20230705-181613 op=CREATE
level=info ts=2023-07-05T10:16:13.144664458Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log op=CREATE
level=info ts=2023-07-05T10:16:23.174391523Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log.20230705-181623 op=CREATE
ts=2023-07-05T10:16:23.174424625Z caller=log.go:168 level=info msg="Re-opening moved/deleted file /var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log ..."
ts=2023-07-05T10:16:23.174452889Z caller=log.go:168 level=info msg="Waiting for /var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log to appear..."
level=info ts=2023-07-05T10:16:23.174535974Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log op=CREATE
level=info ts=2023-07-05T10:16:33.205825404Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log.20230705-181633 op=CREATE
level=info ts=2023-07-05T10:16:33.205986155Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log op=CREATE
level=info ts=2023-07-05T10:16:43.237606654Z caller=filetargetmanager.go:179 msg="received file watcher event" name=/var/log/pods/renderelf_renderelf-frontend-bf89b465c-qwg79_6379d476-a439-499f-99d8-4a1c442349e9/renderelf-frontend/0.log.20230705-181643 op=CREATE
2

I solved my problem Promtail does not increase positions, but set positions zero after a few hours. And then not report logs anymore. · Issue #9864 · grafana/loki · GitHub.

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.