Loki Retention per Stream not working as expected

Grafana Loki
1

I did my best to set up retention per stream.
General idea is to have streams that ca be selected by a label ‘protocol’ which contains the value for the retention of that stream, e.g 3d for 3 days. I have a Minio as S3 storage for the Loki-Data.
For testing I set up a pod that writes appropriate logs (“protocol-log-3d”, “protocol-log-7d”). The streams are selected correctly within Grafana explore-view with Loki source.

Current result: selector: ‘{protocol=“protocol-log-3d”}’ is not deleting log entries older than 3 days (stream retention), nor older than 5 days (global retention)
Expected: Log entries older than retention should be deleted.

Loki runs in a Kubernetes Cluster, Deployment via Helm, image: repository: grafana/loki tag: 2.3.0

I would be really happy to get some opinion (bug or miscofiguration), I can also provide further details if needed.

In case it is a useful hint: while testing different configurations I saw stream retention working (I think during outcommenting and incommenting global retention) but woasnt able to get it workig consistently

Current configuration (relevant parts taken from the loki-secret within the Cluster):

auth_enabled: false
chunk_store_config:
  max_look_back_period: 0s
compactor:
  compaction_interval: 10m
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  retention_enabled: true
  shared_store: s3
  working_directory: /data/loki/boltdb-shipper-compactor
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  max_transfer_retries: 0
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 120h
  retention_stream:
  - period: 72h
    priority: 5
    selector: '{protocol="protocol-log-3d"}'
  - period: 168h
    priority: 5
    selector: '{protocol="protocol-log-7d"}'
schema_config:
  configs:
  - from: "2020-07-01"
    index:
      period: 24h
      prefix: index_
    object_store: aws
    schema: v11
    store: boltdb-shipper
server:
  http_listen_port: 3100
storage_config:
  aws:
  <S3/Minio-Configuration>
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: s3
  filesystem:
    directory: /data/loki/chunks
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Any help appreciated, thanks in advance!

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.