Grafana version 8 (docker image)
I’m going to apologize in advance for not providing source code however the situation I’m seeking help on is on an airgapped system with some fairly stringent rules for moving data of any kind back to internet connected machines. The setup is a bit too complex to even attempt to retype all relevant configurations by hand. Aside, I’m not really looking for help debugging my setup so much as wanting to understand how an aspect of Grafana works so that I can figure out how my setup should be designed in the first place. Hopefully the rest of this matches your template closely enough.
** What are you trying to achieve **
We are trying to setup an instance of Grafana which can access datasources but only for users with the correct credentials.
To elaborate: within the airgapped system I mention there is a second internal system. The internal system is mostly isolated but does have a reverse proxy that can be used to access the services within the system. This reverse proxy also handles authentication to keep access to the internal system restricted. The services within the internal system can communicate with each other freely.
We would like a user facing Grafana that can expose data held by services within the the internal system but only to those who can authenticate with the internal system to begin with. We ran into an issue creating a Grafana instance within the internal system in that we discovered that both the main reverse proxy and Grafana both use a header called ‘Authorization’ to determine access and they are both expecting different things within this header therefore, any communication that can authenticate with the reverse proxy cannot authenticate with Grafana and vice versa. We did look into trying to get Grafana to authenticate in the fashion provided by the main reverse proxy but that line wasn’t looking promising.
** How are you trying to achieve it **
The idea we are currently attempting is to put a second reverse proxy behind the main one and load it with an API key for Grafana. For the users we’d like to show this information we’ll provide them the path to the internal reverse proxy and ideally their authentication will get them through the outer reverse proxy and routed to the inner one which will take their request, swap authorization headers and forward it to Grafana via the internal networks communication and then relay back the dashboards and everything else Grafana serves up.
** What happened **
To some extent this appears to work. The headers get swapped and the API key we provided is validated. Every request we send reaches Grafana and returns with a 200 response so we’re able to retrieve the fonts, stylesheets, icons, and some of the javascript without any difficulty however after taking a bit of time attempting to load it shows the “If you are seeing this Grafana failed to load its application files” error with a list of suggestions. We’re fully willing to believe that either of our proxy configurations might be causing issue but if so we aren’t sure how.
We compared the internal Grafana to a second Grafana instance that we have running outside the internal system and we can see that when one first accesses the external Grafana that all the same requests we saw on the internal Grafana are made but there are several others in addition (mostly more javascript). I presume that these are the application files that Grafana is missing. I am not sure how Grafana can be successfully locating some of it’s files through the various proxies and whatnot but unable to find others that look like they must be in basically the same location (going off the requests).
** What did you expect to happen **
I was hoping (though not exactly expecting) that if the proxies were properly set up to pass the request that users would be able to load the internal Grafana in their browsers and access data like they do the external one.
To restate the main question: How does Grafana attempt to locate it’s frontend files and how does it make the determination that it can’t find them to display that error?
** Did you receive any errors in the Grafana UI or in related logs **
I already mentioned the one shown on the UI. I did not see any errors in the logs for Grafana, or either proxy.
** Did you follow any online instructions **
I have spent quite a bit of time looking in the Grafana documentation both for this current solution and while working on other solutions mentioned throughout this request. The most relevant to this solution would probably be the reverse proxies however I wouldn’t say I was following instructions on those as neither reverse proxy I was working with follows the form of the proxies listed on that page.
As a note: The vast majority of the contstraints of the system exist for a reason and are beyond my control. I do however have full control over the second reverse proxy and so if there is some way the request can be worked to get through I should be able to set it to do so.
–A potential alternate solution would be if there was some way to configure Grafana to use the users authentication when it makes requests to the datasources (so we could configure a datasource to have it look through the main reverse proxy if being accessed by a user with the correct creds) however I couldn’t figure out how to do this while looking through the documentation.
Thanks in advance for the help.