I have a self-hosted cluster with 2 Grafana servers running on Windows. The cluster runs behind a httpd reverse proxy so users get directed to one or the other Grafana instance. The servers are running Grafana 8.3.3. The setup relies on the Grafana Live mode to stream live data to the dashboards via websockets. The httpd is set up to proxy HTTP and WebSockets.
There are two independent publishing processes that push the data to Grafana using the HTTP Live API. Each of these publishers pushes data to only one of the Grafana instances (see diagram).
This setup works in the ideal conditions, but I have an issue when one of the data publisher processes fails. The Grafana instance associated with the publisher is still running, so any browsers connected to this instance do not notice any problem. But the Grafana instance is no longer receiving any data, so the dashboard in the browser is no longer updated and the end-user does not see any live data.
I can resolve this situation by stopping the Grafana server associated with the faulty publisher. The browser detects the broken websockets and re-establishes a connection, which the proxy will send to the other Grafana server.
Question: I would like to ask if there is a way to instruct Grafana (via API?) to terminate the existing Live websocket connections. In other words, is there a way to push existing browser websocket connections from one Grafana cluster instance to another without having to stop Grafana.