Loki S3 Storage/Helm Chart

Grafana Loki
1

Using the helm chart here - Install the Helm Chart | Grafana Loki documentation and setting up S3 storage, I would expect to see objects in the S3 buckets on my minio install, but the buckets are empty. I see the helm chart deploys PVCs in the cluster for read/write (6 total) 10GB each. I can’t find mention of this but does it write to the PVCs first and then flush to S3? How often does Loki do this?

2

Hi @tman5 ,

could you post your Loki config. That might reveal what is going on. Also if there are any logs for e.g. the ingester.

I think the flush to your backend storage is controlled by -ingester.max-chunk-age, probably by some other settings as well.

3

Hi @b0b We’re using minio as the S3 backend on-prem. I see these errors in the backend pods the Loki chart creates:

level=error ts=2023-04-18T19:26:19.706097952Z caller=index_set.go:103 table-name=loki_index_19464 user-id=fake msg="failed to initialize table loki_index_19464, cleaning it up" err="AuthorizationHeaderMalformed: The authorization header is malformed; the region is wrong; expecting 'us-east-1'.\n\tstatus code: 400, request id: 17571E01FB44CFF4, host id: "

Here is the config:

  auth_enabled: false
  storage:
    type: s3
    bucketNames:
      admin: loki-admin
      chunks: loki-chunks
      ruler: loki-ruler
    s3:
      s3: https://loki:loki@minio.internal.com.:443
      s3ForcePathStyle: true
      insecure: false
      region: ""
      http_config:
        insecure_skip_verify: true
4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.