Loki flush logs to persistent storage on shutdown

Grafana Loki
1

Hello

We are evaluating loki as a logging solution to handle TB’s of logs daily to go into S3.
For the trial, we used a docker compose setup like below. The random logger prints log lines and those do appear in grafana, but we notices that if we shutdown loki normally (sudo docker-compose down loki) withing say 30s of starting, the logs in memory are not flushed to disk. Because of our huge volume, we do not want to use WAL.

Is there a way to tell loki ingester to flush all data to persistent storage when the process is shutdown (SIGTERM).

services:

  loki:
    container_name: loki
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/loki-local-config.yaml
    volumes:
      - ./:/etc/loki
      - ./loki-data:/tmp

  grafana:
    container_name: grafana
    depends_on:
      - "loki"
    image: grafana/grafana:latest
    ports:
      - 80:3000
    volumes:
      - ./grafana-data:/var/lib/grafana # grafana-data has grafana persistent data

  random-logger:
    container_name: random-logger
    depends_on:
      - "loki"    
    logging:
      driver: loki
      options:
        loki-url: "http://localhost:3100/loki/api/v1/push"
    #image: chentex/random-logger:latest
    #command: 100 400 10
    image: bash
    command: bash /tmp/seq.sh
    volumes:
      - ./seq.sh:/tmp/seq.sh

seq.sh is this

count=1
while (( 1 )); do
        echo "message id [$count]"
        count=$(( count + 1 ))
        sleep 1
done

Loki config is this

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
  - from: 2020-05-15
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 168h

storage_config:
  boltdb:
    directory: /tmp/loki/index

  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

regards
aniruddha

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.