Cannot save chunks to S3 bucket

Grafana Loki
1

I am trying to connect my Loki instance to S3 bucket so I can save log chunks in the cloud. Here is the config that I am applying:

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: s3
  aws:
    bucketnames: fr-loki-log-storage
    endpoint: s3.eu-west-1.amazonaws.com
    region: eu-west-1
    access_key_id: <access_key_id>
    secret_access_key: <secret_access_key>
    sse_encryption: true

I have the following policy for the bucket:

...
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
...

but I keep getting the following error:

level=error ts=2022-02-28T17:31:53.967586933Z caller=flush.go:221 org_id=53299a67678118040c008cac msg="failed to flush user" err="open <bucket_hash>: permission denied"

Any idea what I might be missing?

2

… more to the point I have successfully connected my ruler to an S3 bucket successfully using the following config:

ruler:
  storage:
    type: s3
    s3:
      bucketnames: fr-loki-ruler-storage
      region: eu-west-1
      access_key_id: <access_key_id>
      secret_access_key: <secret_access_key>
      insecure: false
      sse_encryption: true
      http_config:
        idle_conn_timeout: 90s
        response_header_timeout: 0s
        insecure_skip_verify: false
      s3forcepathstyle: false
  rule_path: /tmp/loki/rules-temp
3

Hi,

this is the config I ended up using for myself. I tried many different ways…

      aws:
        s3: s3://${LOKI_S3_ACCESS_KEY_ID}:${LOKI_S3_SECRET_ACCESS_KEY}@eu-west-1/my_s3_bucket

If I recall correctly, the secret key can not contain the character / or it wont work.

Hope that helps :slight_smile:

4

Hey b0b. Thanks for reaching out. I think I have already tried that and still the same result. Maybe I need remove some of the other defined properties, like bucketnames or region (since they are defined in that URL).

Also … are there any other characters that would need escaping apart from ‘/’?

5

That is the whole aws: part. I would remove bucketnames, endpoint, region, access_key_id and secret_access_key.

And for escaping the /, again, if I remember correctly, I was not able to escape it, I generated a new key until I got one without a forward slash.

6

It turned out to be a different problem. The object store for the boltd-shipper, had to be specified to be in the same storage as the log chunks storage.

However your suggestion was helpful to determine that I had the correct connection configuration.

7

@georgikavalovexterna mind posting a sanitized version of your config. Having the same permission denied issue in EKS trying to use OIDC roles and thinking it might be due to s3 somehow even though it works with the single node deploy of loki

8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.