Delete request stuck at status received

[danielgoepp]

I have a bad feeling I will feel really stupid when someone tells me what I’m doing wrong.

I have pretty simple out of the box default install running on Kubernetes (k3s). It works great. I have configured the compactor with the following settings:

compactor:
  retention_enabled: true
  retention_delete_delay: 3m
  compaction_interval: 3m
  delete_request_cancel_period: 3m
  deletion_mode: filter-and-delete

And then I hit it with the following command to delete some logs:

curl -g -X POST 'http://loki.goepp.net/loki/api/v1/delete?query={app_kubernetes_io_name="traefik"}&start=1672358400&end=1672401459'

I can see the request created:

curl -X GET 'http://loki.goepp.net/loki/api/v1/delete'
[{"request_id":"5c4e244c","start_time":1672358400,"end_time":1672401459,"query":"{app_kubernetes_io_name=\"traefik\"}","status":"received","created_at":1672401472.78}]

But then nothing. I did this a few days ago and nothing ever happened. I deleted the request and submitted again today, still nothing.

I’m missing something obvious aren’t I? Ideas?

Thanks!

[panchambaruah]

Have you checked this?

Configuration for deletes has changed Upgrading | Grafana Loki documentation

[danielgoepp]

Thank you!

No, I had not read that since I was not upgrading. I had noticed in the documentation that it is deprecated in the compactor section, so I moved it to limits_config section, but that didn’t affect things. Also, I was just setting it to the default. I don’t run multi-tenant. In the limits_config section is says allow_delete is deprecated too and to use deletion_mode per-tenant instead. This is not clear to me what I should do then, but I dumped the config and saw that allow_delete was false. I forced it to true and it works! But, I suspect I’m solving this problem wrong with a deprecated method. I guess I need to investigate tenants more.