Loki - Log deletion with line filtering not working

Grafana Loki
1

Summary

What am I doing
Deleting logs from Loki via compactor API endpoint

What I want to achieve
Delete logs based on stream and line filters

My current results
I can only delete logs based on stream filters, but not line

Loki version: 2.6.1

Details

Hello!
I have a quite big deployment of Loki with almost all components, for storage I use S3. From time to time, I need to delete some logs that match specific streams and line filters. Whenever I include line filters in a query Loki does not delete any of the logs. I’ll share more details below.

Configurations

Relevant part of Loki config

common:
  compactor_address: loki-distributed-compactor.loki.svc.cluster.local:3100

compactor:
  compaction_interval: 10m
  delete_request_cancel_period: 15m
  deletion_mode: filter-and-delete
  retention_delete_delay: 2h
  retention_enabled: true
  shared_store: s3

ingester:
  chunk_idle_period: 30m
  chunk_retain_period: 1m

query_range:
  cache_results: true
  results_cache:
    cache:
       enable_fifocache: false
       memcached_client:
         addresses: dns+loki-distributed-memcached-frontend.loki.svc.cluster.local:11211

storage_config:
  boltdb_shipper:
     cache_ttl: 24h
     shared_store: s3

Relevant part of Overrides config

overrides:
  main:
     allow_deletes: true

Relevant part of the gateway (Nginx) config

location = /loki/api/v1/delete {
    set $loki_api_v1_delete_backend http://loki-distributed-compactor.loki.svc.cluster.local;
    proxy_pass       $loki_api_v1_delete_backend:3100$request_uri;
    proxy_http_version 1.1;
}

Story

I have a specific service that I trigger to create logs with the label {facility: "archilogen"}, I want to delete logs only containing the message “archilogen-warn”. I first generate logs, then wait for 30m+ (maybe even multiple hours) so chunks become idle and are sent to S3. After this I send the following API request to the compactor:

curl -I -X POST -H 'X-scope-OrgID: main' -g 'http://xxx/loki/api/v1/delete?query={facility="archilogen"}|="archilogen-warn"'
HTTP/1.1 204 No Content

I can see that request was successful:

curl -H 'X-scope-OrgID: main' 'http://xxx:3020/loki/api/v1/delete' | jq '.[] | select(.status=="received")'

{
  "request_id": "7cdae6e5",
  "start_time": 0,
  "end_time": 1667381172.093,
  "query": "{facility=\"archilogen\"}|=\"archilogen-warn\"",
  "status": "received",
  "created_at": 1667381172.093
}

Theoretically according to my config they should be deleted in 15m, Yet after many hours, non of the logs are deleted.

On the other hand, if I issue the following request, I’ll definitely see that there are matching logs

curl -H 'X-scope-OrgID: main' -g 'http://xxx:3020/loki/api/v1/query_range?query={facility="archilogen"}|="archilogen-warn"&start=1667281172' | jq

And what’s even more strange, If I set query= parameter to simply {facility="archilogen"} all logs will be deleted within 15-30m

One thing I suspected initially was caching, but in this case similar would happen when using label filters only.

Would you have any suggestions?

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.