Unable to PUT when using s3 configuration with loki chart

Grafana Loki
1

Hello,

I’m using the simple “loki” chart version 3.2.1 from Grafana Community Kubernetes Helm Charts | helm-charts (which I believe is the chart moved to loki/production/helm/loki at main · grafana/loki · GitHub, formerly the simply scalable one) and I’m a bit unclear in how to setup correctly the s3 storage.

I have the following values:

        monitoring:
          selfMonitoring:
            enabled: false
            lokiCanary:
              enabled: false
            grafanaAgent:
              installOperator: false
          rules:
            enabled: false
          alerts:
            enabled: false
          serviceMonitor:
            enabled: false
        loki:
          auth_enabled: false
          commonConfig:
            replication_factor: 1
          storage:
            type: 's3'
            bucketnames:
              admin: ae-loki-logging-admin
              chunks: ae-loki-logging-chuncks
              ruler: ae-loki-logging-ruler
          s3:
            endpoint: https://s3.eu-west-1.amazonaws.com
            region: eu-west-1
            secretAccessKey: <secret>
            accessKeyId: <key>
        read:
          replicas: 1
          nodeSelector:
            app: infra
          tolerations:
          - key: "app"
            operator: "Equal"
            value: "infra"
            effect: "NoSchedule"
          persistence:
            size: 16Gi
        write:
          replicas: 2
          nodeSelector:
            app: infra
          tolerations:
          - key: "app"
            operator: "Equal"
            value: "infra"
            effect: "NoSchedule"
          persistence:
            size: 32Gi
        gateway:
          nodeSelector:
            app: infra
          tolerations:
          - key: "app"
            operator: "Equal"
            value: "infra"
            effect: "NoSchedule"
          image:
            tag: 1.23-alpine

Now, when I deploy this and send things to Loki, what I see in write logs is this

level=error ts=2022-10-07T16:57:29.644823039Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: RequestError: send request failed\ncaused by: Put \"https://chunks.s3.dummy.amazonaws.com/fake/93b70bc01000c7c0/183b31afd82%3A183b31bd4a7%3Aceeba107\": dial tcp: lookup chunks.s3.dummy.amazonaws.com on 10.100.0.10:53: no such host"

No matter what I specify for endpoint of bucketnames, the url is ‘chunks.s3.dummy.amazonaws’. I’ve looked at various examples and so on, but I’m left unclear as to the following:

  • what should the s3.endpoint be (as per values file in the repo)
  • what should s3.s3 be (as per values file in the repo)

I’m doing something wrong but not entirely sure what

2

Did you find the solution/workaround on this issue? I also faced the same problem

1 Like
3

Took my many hours to figure this out. Documentation is definitely in a terrible state.

I am using loki (4.5+) and promtail (6.8.2) as subcharts to my own helm chart and this is my configuration that finally got it working.

# Values for loki subchart
loki:
  tenant_id: "my-cluster"
  loki:
    auth_enabled: true
    querier:
      multi_tenant_queries_enabled: false
    storage:
      type: s3
      bucketNames: 
        chunks: "<loki-bucketname>"
        ruler: "<loki-bucketname>"
        admin: "<loki-bucketname>"
      s3:
        s3: s3://<loki-bucketname>
        # Endpoints: https://docs.aws.amazon.com/general/latest/gr/s3.html
        endpoint: ""
        region: "eu-west-1"
        secretAccessKey: ""
        accessKeyId: ""
        s3ForcePathStyle: false
        insecure: false
        sse_encryption: true
        http_config: {}

# Values for promtail subchart
promtail:
  config:
    clients:
      - url: http://loki-gateway/loki/api/v1/push
        tenant_id: "my-cluster"

Then in my chart templates I have a grafana configmap that adds loki as a datasource

{{- if .Values.setup.loki }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasource-loki
  labels:
    grafana_datasource: "1"
data:
  datasource.yaml: |-
    apiVersion: 1
    datasources:
      - name: loki
        type: loki
        url: "http://loki-gateway"
        jsonData:
          httpHeaderName1: 'X-Scope-OrgID'
        secureJsonData:
          httpHeaderValue1: '{{ .Values.loki.tenant_id }}'
{{- end }}

Most likely the part of this configuration that interest you will be this one, as the tenant and auth configuration is specific to me to be able to query logs using one tenant ID per cluster and having one central loki that can query all the tenant ids.

loki:
    storage:
      type: s3
      bucketNames: 
        chunks: "<loki-bucketname>"
        ruler: "<loki-bucketname>"
        admin: "<loki-bucketname>"
      s3:
        s3: s3://<loki-bucketname>
        # Endpoints: https://docs.aws.amazon.com/general/latest/gr/s3.html
        endpoint: ""
        region: "eu-west-1"
        secretAccessKey: ""
        accessKeyId: ""
        s3ForcePathStyle: false
        insecure: false
        sse_encryption: true
        http_config: {}
4

Value of endpoint for eu-west-1 for example would be the following:
s3-accesspoint.eu-west-1.amazonaws.com

5

I am not using secretAccessKey or accessKeyId so they could be left null as per default.
In the bucket names should i use the same bucket for chunks, ruler and admin?
What endpoint should I be using?

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.