S3 + DynamoDB = error querying storage" err="QueryPages error: table=loki_XXX: RequestCanceled: request context canceled\ncaused by: context canceled"

Grafana Loki
1

Hi,

I’m trying to set up PoC for Loki in AWS, running many other instances on local storage pretty sucessfully.

I was struggling a lot to get IAM correct, but now I see tables in dynamodb being created and data coming into S3. Config snippet:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 0.0.0.0
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
    - from: 2020-01-01
      store: aws
      #object_store: s3
      schema: v11
      index:
        prefix: loki_
        period: 168h

storage_config:
  aws:
    s3: s3://region/bucket_name
    dynamodb:
      dynamodb_url: dynamodb://region

table_manager:
  retention_deletes_enabled: true
  retention_period: 504h
  index_tables_provisioning:
    enable_ondemand_throughput_mode: true
    enable_inactive_throughput_on_demand_mode: true

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

Unfortunately, when I’m trying to add datasource in Grafana, I’m getting errors:

loki_1      | level=error ts=2021-05-11T21:20:46.432554487Z caller=chunk_store.go:543 org_id=fake traceID=714ce7a2197fa7d4 msg="error querying storage" err="QueryPages error: table=loki_2679: RequestCanceled: request context canceled\ncaused by: context canceled"
loki_1      | level=warn ts=2021-05-11T21:20:46.432681015Z caller=logging.go:71 traceID=714ce7a2197fa7d4 msg="GET /loki/api/v1/label?start=1620767437053000000 (500) 9.998483623s Response: \"QueryPages error: table=loki_2679: RequestCanceled: request context canceled\\ncaused by: context canceled\\n\" ws: false; Accept: application/json, text/plain, */*; Accept-Encoding: gzip, deflate, br; Accept-Language: en-GB,en-US;q=0.9,en;q=0.8; Sec-Ch-Ua: \"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"; Sec-Ch-Ua-Mobile: ?0; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin; User-Agent: Grafana/7.5.5; X-Forwarded-For: 3.1.212.129, 127.0.0.1, 127.0.0.1; X-Grafana-Nocache: true; X-Grafana-Org-Id: 1; X-Real-Ip: 3.1.212.129; "
loki_1      | level=error ts=2021-05-11T21:21:09.156881146Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:21:09.156960693Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:21:19.157084455Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=info ts=2021-05-11T21:21:29.154660587Z caller=table_manager.go:324 msg="synching tables" expected_tables=4
loki_1      | level=error ts=2021-05-11T21:21:39.156841084Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:21:39.156833134Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:21:49.157040397Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:22:09.156848093Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:22:09.156848135Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:22:19.157057025Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:22:39.156841956Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:22:39.156876196Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
loki_1      | level=error ts=2021-05-11T21:22:49.157073855Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="RequestCanceled: request context canceled\ncaused by: context deadline exceeded"

I checked dynamodb IAM permissions, seems to be fine. Where should I look for the issue?

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.