Loki -Queries older that few hours are timing out (S3 backend)

Grafana Loki
1

I am new with Loki and have not read this fully; But I am running Loki + Grafana in a AWS EKS cluster with nothing else. Queries up to 3 hours are coming fast. Anything else say 6 hours or 24 hours is timing out. Could not see any particular error logs in any of the pods in loki namespace. S3 is default AWS configuration. Not sure what I am doing wrong. I know read via S3 will be slow; but I cannot imagine it to be so slow that it times out. Is there any major problem with the settings belpow

loki:
  auth_enabled: false
  commonConfig:
    path_prefix: /var/loki
    replication_factor: 1
  compactor:
    apply_retention_interval: 1h
    compaction_interval: 5m
    retention_delete_worker_count: 500
    retention_enabled: true
    shared_store: s3
    working_directory: /data/compactor
  config:  
      schema_config:
        configs:
        - from: 2020-05-15
          store: boltdb-shipper
          object_store: s3
          schema: v11
          index:
            period: 24h
            prefix: loki_index_
        
      storage_config:
        aws:
          region: eu-central-1
          bucketnames: alex-loki-test
          s3forcepathstyle: false
        boltdb_shipper:
          shared_store: s3
          cache_ttl: 24h
      limits_config:
          max_entries_limit_per_query: 50000 # default is 5000
  serviceAccount:
    create: true
    name: loki-sa
    annotations:
       eks.amazonaws.com/role-arn: "arn:aws:iam::xxx:role/loki_s3_role"
  write:
     replicas: 2
  read:
    replicas: 1

grafana:
    enabled: true
    sidecar:
      datasources:
        enabled: true
    image:
      tag:  7.5.17
    grafana.ini:
      users:
        default_theme: dark
2

I understood the problem. The rationale is that there is no pagination in Grafana;(Closer look at Grafana's user interface for Loki | Grafana Labs)

“Without paging, you need to make sure what you are looking for is in the result set of the 1000 lines. You do this by filtering.”

So basically without pagination, Grafana is trying to load too much data to Browser and it times out!!

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.