Currently our system running in Kubernetes is using Loki, we would like to use Tempo as well.
When you install Loki you have the option to install Grafana and this gives you a Dashboard where you can get and visualize the logs. However it doesn’t allow you to include Tempo as Datasource.
On the other hand if you independently install Grafana it allows you to include Tempo and lets you query Loki logs but what we want is what Grafana Loki dashboard provides.
Of course the work around is to have two Grafana pods running one for Loki and one for Tempo. This might be frown upon. I’ve been searching but haven’t found the right info or even if that is possible. So the question is if there is a way to combine both dashboards in one UI and pod.
Can you explain a little about how you are installing Grafana with Loki as shown in the first screenshot? Are you using the official helm charts? (Links to the installation scripts would be nice).
Generally, it is possible to add multiple data sources to the same Grafana UI to visualise data. So this is likely a configuration issue that we need to work around. On a regular Grafana dashboard you should be able to add a Tempo datasource as described on this page - Tempo | Grafana Labs
As you can see we are just enabling Grafana which provides the UI I mentioned on the first screenshot of my first post. After running that command loki gets installed as well as Grafana. If you launch the UI running on local:3000 then go to Configuration Add Data Source, you won’t see Tempo as an option to add in. That’s the problem. Furthermore if you try to add a plugin there is none for Tempo. As I mentioned before if in the above command you don’t enable Grafana but install it separately then you have the option to Add Tempo and loki as Data Sources but it doesn’t give you the Loki dashboard that you get when installing loki and enabling Grafana.
As you mention this might be a configuration issue but so far I haven’t found anything that can help me to change it.
Hi @ragamx, I think the issue comes from the Grafana version pinned in those helm charts. The loki-stack helm chart is using v0.6.4, which does not support Tempo as a datasource for Traces. Tempo support was introduced first to Grafana in version v7.3.0-beta1.
The proper solution is to upgrade the Grafana version to a newer one in the helm charts. While that is fixed, you can try using a more recent Grafana version. I don’t think it’s possible to override the version in the charts, but using the Grafana charts should do it.
Yeah, it makes sense what you are saying, the issue is in the Grafana version that comes with the loki-stack chart, which is 6.7.0. As I mentioned in my first post I could use a newer version of Grafana, one that includes Tempo, like 7.5.4. However the current chart loki-stack contains a pre-configured Grafana dashboard that we want. If I install a newer version of Grafana how can I add that dashboard for loki?
Thanks for your input.
P.S. How can I notify Grafana Loki team to get a proper solution?
I’m not sure what specific dashboard you’re referring to, but there is an official loki-mixin, which most likely is the source for any provided dashboard, recording rule and alert in the helm chart.
You can make use of the mixin to have out-of-the-box monitoring for Loki. There are multiple ways of using mixins (see https://monitoring.mixins.dev).
Probably generating config files is the easiest to get started. Just navigate to the loki-mixin folder and run jb install && mixtool generate all mixin.libsonnet (you will need jsonnet and mixtool). That should output the files alerts.yaml and rules.yaml and multiple dashboards inside dashboard_out. You can then import all those files to your grafana instance.
And there is a mixin for Tempo too! Check the instructions to generate dashboards and alerts.
P.S. How can I notify Grafana Loki team to get a proper solution?
I opened a PR to upgrade the Grafana version in the helm chart, but it’s pending approval.
I see what dashboard you’re referring to, thanks! I’m suspecting now that the main issue might be with the datasource. Apologies if you’ve done this already, but in case you’ve not, if you’re deploying Grafana independently, you will need to add Loki as a datasource, as opposed to when using the helm chart (in that case the datasource is added automatically via this file imported to Grafana). You can check this guide on how to do it: Loki in Grafana | Grafana Labs. You will need to do the same for Tempo if you want to use it in the same Grafana instance: Quickstart with Tempo | Grafana Labs.
In any case, I understand that the optimal solution is to upgrade the Grafana version in the Helm chart. I’ll poke around to see if it can be merged.
After upgrading with the latest change the Grafana version gets updated to 7.5.0 but unfortunately this breaks the Loki dashboard from 6.7.0. Now I can see why the Grafana version for Loki didn’t upgrade and stay in that old version for a long time. I think there are issues with that and the new Grafana v7.3.0 and up. Not sure if we should roll back, people might start wondering what happen to that very useful dashboard. Of course you can specify chart version during Loki installation. Chart 2.2.0 has the old version.
Yeah, for the moment I think there is no other option than having two Grafana pods running in Kubernetes one for Loki and one for Temp while all this gets sorted out.
Could you describe what issues you’re encountering with the new version? I did not run into any issues with the new helm chart, so I may have missed it.
Have you tried running all 3 components independently and manually adding the datasources for Tempo and Loki? That should work and enable you to create a panel with Loki logs as the one shown in this comment.
The only missing part that I was referring to is the dashboard shown in the first screenshot of my first post. Are you able to see it? I guess the old Loki was given that by default when adding a Dashboard Query, with the new Grafana you may need to create it but in a different way to get the same info, that’s all, maybe not a biggie. I think it should be OK. At least with the new version I can see all the Datasources.
