Grafana cloudwatch integration is very slow + how to set up dashbaord?

disabled websockets to get logs to load. a simple query takes 1min to load.

  • how can I fix this issue? haven’t seen anything online yet

  • I want to allow people to search queries or for services, ideally without having to use query editor but don’t want them changing the panels with EDIT access. Is there a way?

  • when someone clicks on the logs, grafan re-directs to AWS console. is there a way to view entire log message in Grafana without having to export to CSV?


some stats


  • What Grafana version and what operating system are you using?
    Grafana version: version: “8.1.3”

  • What are you trying to achieve?
    create a dasbhboard with usable logs from cloudwatch. Queries are too slow (1min for a simple query)

  • How are you trying to achieve it?
    i’m not sure where to start

  • What happened?
    logs weren’t working. Disabled webhooks → logs now load but VERY slowly

  • What did you expect to happen?
    logs to load quickly as they do on Cloudwatch Log Insights

  • Can you copy/paste the configuration(s) that you are having problems with?


    Stats
    Total request time	53.2 s
    Number of queries	1
    Total number rows	2
    Data source stats
    Bytes scanned	61892959611
    Records scanned	54805339
    Records matched	54732234

    
    {
      "id": 13,
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "type": "table",
      "title": "Recent Logs",
      "datasource": "msm-live",
      "pluginVersion": "8.1.3",
      "fieldConfig": {
        "defaults": {
          "custom": {
            "align": "auto",
            "displayMode": "auto"
          },
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "mappings": [],
          "color": {
            "mode": "thresholds"
          }
        },
        "overrides": []
      },
      "options": {
        "showHeader": true
      },
      "targets": [
        {
          "expression": "fields @timestamp, @message\n| sort @timestamp desc\n| display time, kubernetes.pod_name, log\n| limit 2",
          "id": "",
          "logGroupNames": [
            "/aws/eks/fluentbit-cloudwatch/logs"
          ],
          "namespace": "",
          "queryMode": "Logs",
          "refId": "A",
          "region": "eu-west-1",
          "statsGroups": []
        }
      ]
    }
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No

  • Did you follow any online instructions? If so, what is the URL?
    for log speeds,no.

Hi @musicspaceman

I’m seeing a lot of development on cloudwath plugin between 8.1.3 and now. Can you try upgrading and see if you can reproduce the error? That would be super useful:

15:- **CloudWatch:** Remove error message when using multi-valued template vars in region field. [#45886](https://github.com/grafana/grafana/pull/45886), [@sunker](https://github.com/sunker)
22:<!-- 8.4.3 END -->
36:<!-- 8.4.2 END -->
43:- **Cloudwatch:** Add support for AWS/PrivateLink\* metrics and dimensions. [#45515](https://github.com/grafana/grafana/pull/45515), [@szymonpk](https://github.com/szymonpk)
51:<!-- 8.4.1 END -->
74:- **CloudWatch:** Add Data Lifecycle Manager metrics and dimension. [#43310](https://github.com/grafana/grafana/pull/43310), [@ilyastoli](https://github.com/ilyastoli)
75:- **CloudWatch:** Add Missing Elasticache Host-level metrics. [#43455](https://github.com/grafana/grafana/pull/43455), [@dhendo](https://github.com/dhendo)
76:- **CloudWatch:** Add all ElastiCache Redis Metrics. [#43336](https://github.com/grafana/grafana/pull/43336), [@siavashs](https://github.com/siavashs)
77:- **CloudWatch:** Add new AWS/ES metrics. [#43034](https://github.com/grafana/grafana/pull/43034), [@sunker](https://github.com/sunker)
78:- **Cloudwatch:** Add syntax highlighting and autocomplete for "Metric Search". [#43985](https://github.com/grafana/grafana/pull/43985), [@sarahzinger](https://github.com/sarahzinger)
91:- **CloudWatch:** Fix MetricName resetting on Namespace change. [#44165](https://github.com/grafana/grafana/pull/44165), [@yaelleC](https://github.com/yaelleC)
92:- **Cloudwatch :** Fixed resetting metric name when changing namespace in Metric Query. [#44612](https://github.com/grafana/grafana/pull/44612), [@yaelleC](https://github.com/yaelleC)
106:<!-- 8.4.0 END -->
134:<!-- 8.3.7 END -->
162:<!-- 8.3.6 END -->
163:<!-- 8.4.0-beta1 END -->
172:<!-- 8.3.5 END -->
210:<!-- 8.3.4 END -->
224:- **CloudWatch:** Allow queries to have no dimensions specified. [#42800](https://github.com/grafana/grafana/pull/42800), [@sunker](https://github.com/sunker)
225:- **CloudWatch:** Fix broken queries for users migrating from 8.2.4/8.2.5 to 8.3.0. [#42611](https://github.com/grafana/grafana/pull/42611), [@sunker](https://github.com/sunker)
226:- **CloudWatch:** Make sure MatchExact flag gets the right value. [#42621](https://github.com/grafana/grafana/pull/42621), [@sunker](https://github.com/sunker)
237:<!-- 8.3.3 END -->
244:<!-- 8.3.2 END -->
252:<!-- 8.3.1 END -->
264:- **CloudWatch:** Add support for AWS Metric Insights. [#42487](https://github.com/grafana/grafana/pull/42487), [@sunker](https://github.com/sunker)
277:<!-- 8.3.0 END -->
287:- **CloudWatch:** Add missing AWS/Events metrics. [#42164](https://github.com/grafana/grafana/pull/42164), [@n2N8Z](https://github.com/n2N8Z)
353:<!-- 8.3.0-beta2 END -->
374:- **CloudWatch Logs:** Add retry strategy for hitting max concurrent queries. [#39290](https://github.com/grafana/grafana/pull/39290), [@aocenas](https://github.com/aocenas)
375:- **CloudWatch:** Add AWS RoboMaker metrics and dimension. [#41450](https://github.com/grafana/grafana/pull/41450), [@ilyastoli](https://github.com/ilyastoli)
376:- **CloudWatch:** Add AWS Transfer metrics and dimension. [#41168](https://github.com/grafana/grafana/pull/41168), [@ilyastoli](https://github.com/ilyastoli)
432:<!-- 8.2.7 END -->
447:<!-- 8.2.6 END -->
448:<!-- 8.3.0-beta1 END -->
475:<!-- 8.2.5 END -->
482:<!-- 8.2.4 END -->
490:<!-- 8.2.3 END -->
521:<!-- 8.2.2 END -->
533:<!-- 8.2.1 END -->
545:- **CloudWatch Logs:** Skip caching for log queries. [#39860](https://github.com/grafana/grafana/pull/39860), [@aocenas](https://github.com/aocenas)
565:<!-- 8.2.0 END -->
578:- **CloudWatch:** Introduced new math expression where it is necessary to specify the period field. [#39458](https://github.com/grafana/grafana/pull/39458), [@sunker](https://github.com/sunker)
618:<!-- 8.2.0-beta2 END -->
637:- **CloudWatch Logs:** Add link to X-Ray data source for trace IDs in logs. [#39135](https://github.com/grafana/grafana/pull/39135), [@aocenas](https://github.com/aocenas)
638:- **CloudWatch Logs:** Disable query path using websockets (Live) feature. [#39231](https://github.com/grafana/grafana/pull/39231), [@aocenas](https://github.com/aocenas)
639:- **CloudWatch/Logs:** Don't group dataframes for non time series queries. [#37998](https://github.com/grafana/grafana/pull/37998), [@aocenas](https://github.com/aocenas)
640:- **Cloudwatch:** Migrate queries that use multiple stats to one query per stat. [#36925](https://github.com/grafana/grafana/pull/36925), [@sunker](https://github.com/sunker)
711:<!-- 8.2.0-beta1 END -->
719:<!-- 8.1.8 END -->
730:<!-- 8.1.7 END -->
738:<!-- 8.1.6 END -->
748:<!-- 8.1.5 END -->
769:<!-- 8.1.4 END -->
793:<!-- 8.1.3 END -->
828:<!-- 8.1.2 END -->
835:- **CloudWatch Logs:** Fix crash when no region is selected. [#37639](https://github.com/grafana/grafana/pull/37639), [@aocenas](https://github.com/aocenas)
838:<!-- 8.1.1 END -->