How Grafana Loki retention works

[dorinand]

I would like to set up retention for logs stored in Grafana Loki and I am not sure how the retention really works. There is more period time to configure and from documentation, it is not so clear to understand how whole process of retention works.

This is my Loki configuration:

common:
  instance_interface_names:
    - "lo"
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_interface_names:
      - "lo"
    kvstore:
      store: inmemory

compactor:
  working_directory: /loki/compactor
  compaction_interval: 5m
  retention_enabled: true
  retention_delete_delay: 5m
  delete_request_cancel_period: 1m

limits_config:
  retention_period: 10m

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
      chunks:
        prefix: chunk_
        period: 168h

Compactor should remove old data based on this configuration. This is how I understand config file:

• compactor.compaction_interval - every 5 minutes is run compactor process
• compactor.retention_enabled - enable to apply retention policy
• compactor.retention_delete_delay - this is the time Compactor is waiting to delete chunks after retention_period exceed.
• compactor. delete_request_cancel_period - allow cancellation of delete request made by API. I think, not used in my case at all.
• limits_config. retention_period - this is time after which data should be deleted. According to doc - The minimum retention period is 24h.

According to the configuration, I would expect to have logs for last 15 minutes:

• 0:00 - first log arrives
• 0:05 - compactor is running and because of retention 10 min, no logs are meant to be deleted
• 0:10 - compactor is running again and because of retention 10 min, my first log should be assign for deletion. Because of retention delay 5min, it is still not deleted at this moment.
• 0:15 - compactor is running for third time. From the second run, there has been log to be deleted and retention delay 5 min has expired, that means this log can be deleted now.

This is just an example of ideal time schedule. I understand there is async processes and timing is not so precise so lets say, according to my logic, this could be done in 30-40 minutes.

I am running this configuration and I can see logs for last 2,5 hours:

First log at 20:43 and last one ad 23:00. This is quite long according to my estimation. I do not want to use it for this short period of time but how can I estimate log retention for example for 3 days, 1 week or 2 months?

Questions:

1. How is retention calculated and processed?
2. Do I understand Loki configuration parameters properly?
3. How does retention differs from chunk.period and index.period under schema_config.config?

[ottovsky]

I am interested in knowing how retention exactly works as well, hopefully someone from loki team can shed some light on it.

[ericleecnl996]

@dorinand

/loki/chunks

If you look at the /loki/chunks directory, you will be surprised to find that the log files are not deleted, causing the disk space to become full. I’m also interested in how this retention works.

[bojanflander]

Can someone please answer these questions?
I am also very interested how to set retention and what exactly mean each of the parameters.