CloudWatch datasource from Grafana docker image in AWS ECS

Grafana Installation
1

Dear All,

I have created a custom docker custom using Grafana 7.3.4, and deployed on AWS ECS. I am accessing through a load balancer (ELB) using http ; ELB port 80 forwarded to port 3000 of ECS instance. I created custom image to include various custom settings e.g. email server which is not possible with standard grafana docker image.

However, when I try to configure a CloudWatch datasoure it is not working. In Graana log, I see this-

t=2021-01-27T18:43:12+0000 lvl=dbug msg=“Querying for data source via SQL store” logger=datasources id=15 orgId=1
t=2021-01-27T18:43:12+0000 lvl=dbug msg=“Authenticating towards AWS with an access key pair” logger=tsdb.cloudwatch region=eu-west-1
t=2021-01-27T18:43:12+0000 lvl=dbug msg=“Successfully created AWS session” logger=tsdb.cloudwatch
t=2021-01-27T18:43:12+0000 lvl=eror msg=“Metric request error” logger=context userId=2 orgId=1 uname=sysadmin error=“failed to call cloudwatch:ListMetrics: RequestError: send request failed\ncaused by: Post “https://monitoring.eu-west-1.amazonaws.com/”: x509: certificate signed by unknown authority” remote_addr=“47.15.1.216, 165.225.124.190”
t=2021-01-27T18:43:12+0000 lvl=eror msg=“Request Completed” logger=context userId=2 orgId=1 uname=sysadmin method=POST path=/api/tsdb/query status=500

It appears it could establish session using AWS Access Key and Secret used; but then call to CloudWatch API errors out due to certificate issue.

How to resolve this? Can configure my grafana docker image to skip SSL verification when calling CloudWatch API?

Thanks in advance for any help.

Regards.
sd

2

yes this will help, pleaes try

3

hi @melrose
how to do that? can you guide please?

4

You can add tls_skip_verify_insecure = true to the [auth.generic_oauth] block

5

Somehow setting tls_skip_verify_insecure = true does not work.

I realized it is working when I run grafana-7.3.4 directly on an EC2 machine, but it stops working when I make a docker image. So basically on EC2 with Amazon linux machine, which is using JDK which trusts the certificate.

I was using ubuntu:latest to build my docker image. I changed to amazonlinux:latest ; and it works. SO that solves problem for now.

However if I want use ubuntu or centos image, how can I bypass this? How can I tell Grafana to ignore ssl certificate errors when adding a CloudWatch datasource - i.e. Post “https://monitoring.eu-west-1.amazonaws.com/”: x509: certificate signed by unknown authority”

6

Why you want to sacrifice TLS security? :hear_no_evil:
Why you just don’t install proper CA certificates to the Docker image, so system/Grafana can create validated/trusted TLS connection. It is trivial command, e.g. Centos: yum install -y ca-certificates :boom:

7

Thank you very much @jangaraj
Yes after installing ca-certificates it works with ubuntu image.
This is right solution. Thanks a lot again.