IMDSv2 is not working from docker

Hi, I’m using Grafana docker hosted on an EC2 instance with IMDSv2 enabled.

While testing with the following curl commands everything looks fine (the token is fake):

$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0  65268      0 --:--:-- --:--:-- --:--:-- 56000
$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
> X-aws-ec2-metadata-token: AQAEAOpL35wR9KpFB7rRxVy7GhnB2mIBGPXNR5tBP4c-f7YWiL1yXA==
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21593
< Content-Type: text/plain
< Accept-Ranges: none
< Last-Modified: Thu, 19 May 2022 11:39:54 GMT
< Content-Length: 337
< Date: Thu, 19 May 2022 12:33:04 GMT
< Server: EC2ws
< Connection: close
< 
ami-id
ami-launch-index
…

Now when I’m trying to add CloudWatch data source to Grafana (inside the docker container) the following message appears (Text is attached as alternative caption for the image):

What am I doing wrong?

Grafana version: 8.5.2
Relevant Attached Policy (although it doesn’t matter): CloudWatchReadOnlyAccess

UPDATE: I’ve discovered that’s it’s not working due to network restrictions.

can you elaborate in case this thread help someone else in the future? thx :+1:

1 Like

Sure, well, docker allows all sorts of networking modes, internal, bridge and host.

When using bridge or host, the container is exposed to everything the machine is exposed to including the internal AWS API (IMDS/v2) via 169.254.169.254.

So, when selecting an internal network for Grafana (although there’s an exposed port) the container cannot reach IMDS, one possible solution is create another network and attach the Grafana container to it or switching to bridge or host mode entirely, in my case it’s not recommended as it exposes promtail and loki without any moderation.

Hoping my explanation makes sense.

1 Like