How to make tag_values query work in Templating with OpenTSDB

Hi,
I’ve read about Templating as a way of making the dashboard more reusable. My first idea was to make dashboards host agnostic and create a host variable. I have OpenTSDB as the datasource so I followed this documentation. Filled in query “tag_values(cpu, host)” and got this error:

Ok. So it’s either cpu or host that might be causing this. The first parameter should be a metrics. So let’s check if the metric is correct. I used metrics(cpu) and got:

  • cpu_time_guest
  • cpu_time_guest_nice
  • cpu_time_idle
  • cpu_time_iowait
  • cpu_time_irq
  • cpu_time_nice
  • cpu_time_softirq
  • cpu_time_steal
  • cpu_time_system
  • cpu_time_user
  • cpu_usage_guest
  • cpu_usage_guest_nice
  • cpu_usage_idle
  • cpu_usage_iowait
  • cpu_usage_irq
  • cpu_usage_nice
  • cpu_usage_softirq
  • cpu_usage_steal
  • cpu_usage_system
  • cpu_usage_user

Ok, let’s use one of these in my query: “tag_values(cpu_usage_user, host)”. No error is shown, but no preview either. The doc states: “If you do not see template variables being populated in Preview of values section, you need to enable tsd.core.meta.enable_realtime_ts in the OpenTSDB server settings. Also, to populate metadata of the existing time series data in OpenTSDB, you need to run tsdb uid metasync on the OpenTSDB server.” I’ve checked the config: "tsd.core.meta.enable_realtime_ts": "true". Maybe it doesn’t work only on preview? So I saved the variable even though it had no preview shown and refreshed the dashboard with Refresh: On Dashboard Load. No results either.

My second guess would be that the tag_name in tag_values(metrics, tag_name) is wrong. So I started experimenting:

  • tag_values(cpu, hostname) => Error: “Templating Template variables could not be initialized: undefined”
  • tag_values(cpu, host) => Error: “Templating Template variables could not be initialized: undefined”
  • tag_values(cpu_usage_user, host) => No error, no values
  • tag_values(cpu_usage_user, hostname) => Error: “Templating Template variables could not be initialized: undefined”

Ok, so the query tag_values(cpu_usage_user, host) doesn’t throw an error, but doesn’t show any results either.

I know for a fact, that both metric cpu/cpu_usage_user exist as well as tag host, but for some reason it doesn’t work.

Please help.

Grafana v4.4.1 (commit: 6a9f8ca)

The solution was to change proxy settings that redirected requests to OpenTSTB.

1 Like

@markmitrushkin could you please elaborate and explain in more detail. I am having the exact same issue, and I am having trouble getting through. Thank you.

  • Add tsd.http.request.cors_domains = * to you opentsdb server conf opentsdb.conf
    cat " tsd.http.request.cors_domains = * " >> /path/to/opentsdb.conf
  • Restart the server of opentsdb.
  • detail for read: HTTP API — OpenTSDB 2.4 documentation