Using Helm I am trying to deploy Prometheus with Grafana with predefined datasources/dashboards. It works well for Kubernetes cluster monitoring, however I have an issue with external server metrics.
I am not sure where the problem is so I am asking here, correct me if I am wrong. Using hxxps://github.com/marcel-dempers/kube-prometheus-grafana with my changes in
values.yaml:
namespace: monitoring
prometheus:
k8-api-server: hxxps://kubernetes.default
service:
name: prometheus
port: 9090
spec:
image: prom/prometheus:v2.5.0
grafana:
service:
name: grafana
spec:
image: grafana/grafana:5.2.0
prometheus:
endpoint: hxxp://prometheus:9090
nfsserver:
endpoint: hxxp://1.2.3.4:8086
and grafana-datasources.configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
name: grafana-datasources
namespace: {{ .Values.namespace }}
data:
kubernetes.json: |-
{
"name": "kubernetes",
"type": "prometheus",
"url": "http://{{ .Values.prometheus.service.name }}:{{ .Values.prometheus.service.port }}",
"access": "proxy"
}
nfs-server-metrics.json: |-
{
"name": "NFS metrics",
"type": "influxdb",
"url": "{{ .Values.grafana.nfsserver.endpoint }}",
"access": "proxy",
"database": "telegraf"
}
Dashboard for NFS metrics is from: hxxps://grafana.com/dashboards/5955
After deployment I have Grafana with two dashboards and two datasources.
As you can see in: hxxps://github.com/marcel-dempers/kube-prometheus-grafana/blob/master/helm/grafana/templates/grafana.deployment.yaml
Datasources are imported before dashboards. However for some reason “Telegraf - system metrics” dashboard can’t talk to datasource saying:
Templating init failed
Datasource named ${DS_INFLUXDB} was not found
datasource dropdown menu is set to NFS metrics, but rest of fields like Server, disk, CPU are empty.
The solution for that is to remove dashboard and import it again manually in Grafana GUI. Any idea why that happen?
(I had to replace http with hxxs, due to limit for new users)