Is EFS a good logs backup option if Loki pod terminated accidentally in EKS Fargate

Grafana Loki
1

I am currently using Loki to store logs generated by my applications from EKS Fargate. Sidecar pattern with promtail is used to scrape logs. Single Loki pod is used and S3 is configured as a destination to store logs. It works nicely as expected. However, when I tested the availability of the logging system by deleting pods, I discovered that if Loki’s pod was deleted, some logs would be missing (range around 20 mins before the pod was deleted to the time the pod was deleted) even after the pod restarted.

To solve this problem, I tried to use EFS as the persistent volume of Loki’ pod, mounting the path /loki. The whole process is followed by this article (New – AWS Fargate for Amazon EKS now supports Amazon EFS | AWS News Blog). But I have got an error from the Loki pod with msg “error running loki” err="mkdir /loki/compactor: permission denied”

Therefore, I have 2 questions in my mind:

Should I use EFS as a solution for log backup in my case?
Why did I get a permission denied inside the pod, any ways to solve this problem?

Thank you!

My Loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  wal:
    enabled: true
    dir: /loki/wal
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 3m       
  chunk_retain_period: 30s    
  max_transfer_retries: 0     
  chunk_target_size: 1048576 

schema_config:
  configs:
    - from: 2020-05-15
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

  aws:
    bucketnames: bucketnames
    endpoint: s3.us-west-2.amazonaws.com
    region: us-west-2
    access_key_id: access_key_id
    secret_access_key:  secret_access_key
    sse_encryption: true

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  compaction_interval: 5m

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 48h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 96h

querier:
  query_ingesters_within: 0

analytics:
  reporting_enabled: false

Deploy.yaml

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    namespace: fargate-api-dev
    name: dev-loki
  spec:
    selector:
      matchLabels:
        app: dev-loki
    template:
      metadata:
        labels:
          app: dev-loki
      spec:
        volumes:
          - name: loki-config
            configMap:
              name: dev-loki-config
          - name: dev-loki-efs-pv
            persistentVolumeClaim:
              claimName: dev-loki-efs-pvc
      containers:
        - name: loki
          image: loki:2.6.1
          args:
            - -print-config-stderr=true
            - -config.file=/tmp/loki.yaml         
          resources:
            limits:
              memory: "500Mi"
              cpu: "200m"
          ports:
            - containerPort: 3100
          volumeMounts:
            - name: dev-loki-config
              mountPath: /tmp
              readOnly: false
             - name: dev-loki-efs-pv
               mountPath: /loki

Promtail-config.yaml

server:
  log_level: info
  http_listen_port: 9080

clients:
  - url: http://loki.com/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  - job_name: api-log
    static_configs:
    - targets:
      - localhost
      labels:
        job: apilogs
        pod: ${POD_NAME}
        __path__: /var/log/*.log
2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.