Help me do something crazy cool but also maybe crazy (?) with templating

nathan3547 · May 20, 2021, 8:52am

I’m currently using netdata to monitor metrics and performance on my nodes in real time. I’ve yet to find anything that tops it for real time measurements.

I’m moving all our stack monitoring and metrics to grafana dashboards. I want to bring netdata realtime dashboards into that. Thanks to netdata custom html templates, and the grafana Text panel I can create this holy love child of the two.

What I want is this: I want to be able to run a query that lists the nodes in my stack filtered by a property, then for each node in the list show a Text panel that has my netdata dashboard in it, passing in the node as a template parameter to inject into the text panel.

I KNOW RIGHT?! Crazy COOL!
Is it even possible?

Yours hopefully,
ndtreviv

svetb · May 20, 2021, 2:33pm

Hi @nathan3547! I think it should be possible. I’m not familiar with netdata though. Are you able to create a Grafana template variable that contains the list of nodes? Once you’ve done that you can set up your text panel to repeat for each value of that variable (and use the variable in the HTML).

nathan3547 · May 20, 2021, 2:47pm

So I was hoping to get the list of nodes via a query against a datasource in grafana - still possible?
Also, I can’t see in the Text panel how to pass variables in… I’m still very much learning

nathan3547 · May 20, 2021, 3:18pm

I’ve worked out how to create dashboard variables. The problem is I’m struggling with the query. The values I want to get come from an elasticsearch datasource, but they’re not store-d, so I can’t create a query to target them.
I can create a table panel that lists them by choosing Raw Data as the metric and applying a couple of transformations, being:

Filter by Name (host.name)
Group By (host.name)

This gives me a set of hostnames from my .monitoring-es-* indices.

I then want those to be the variable values, but I can’t see how to do that in the dashboard Variables query section.
Is this possible, or do I need to change metricbeat to specifically index the hostname as a separate field?

svetb · May 20, 2021, 3:58pm

Thanks for following up! It’s good that you can query the node list via the ElasticSearch datasource. I’m afraid I know very little about that datasource though, so can’t really help with getting the right set of variable values - hopefully someone else on here can. To prompt better responses, maybe you could share some screenshots of what your queries/responses look like.

And indeed, the variable queries lack the comprehensive “slicing and dicing” capabilities that you can use in panels…but hopefully you can get to where you need to.

In the panel itself, you should be able to use $varname inside your HTML, which will interpolate to each of the variable values.

nathan3547 · May 24, 2021, 9:57am

Ok, I figured it out.

First of all I had to modify my filebeat config to include the hostname as a searchable, aggregatable field, then create a new dashboard with a dashboard variable that targeted that field.

Then I created a Text panel with the following content:

<script
  type="text/javascript"
  src="http://${Nodes}:19999/dashboard.js"
></script>
<div class="netdata-chart-row">
  <div
    class="netdata-container-easypiechart"
    style="margin-right: 10px; width: 11%; will-change: transform"
    data-host="http://${Nodes}:19999"
    data-netdata="system.io"
    data-dimensions="in"
    data-chart-library="easypiechart"
    data-title="Disk Read"
    data-width="11%"
    data-before="0"
    data-after="-360"
    data-points="360"
    data-common-units="system.io.mainhead"
    role="application"
  >
    <div
      class="netdata-message icon hidden"
      style="font-size: 11.9px; padding-top: 51.05px"
    >
      <svg
        class="svg-inline--fa fa-chart-area fa-w-16"
        aria-hidden="true"
        data-fa-processed=""
        data-prefix="fas"
        data-icon="chart-area"
        role="img"
        xmlns="http://www.w3.org/2000/svg"
        viewBox="0 0 512 512"
      >
        <path
          fill="currentColor"
          d="M500 384c6.6 0 12 5.4 12 12v40c0 6.6-5.4 12-12 12H12c-6.6 0-12-5.4-12-12V76c0-6.6 5.4-12 12-12h40c6.6 0 12 5.4 12 12v308h436zM372.7 159.5L288 216l-85.3-113.7c-5.1-6.8-15.5-6.3-19.9 1L96 248v104h384l-89.9-187.8c-3.2-6.5-11.4-8.7-17.4-4.7z"
        ></path></svg
      ><!-- <i class="fas fa-chart-area"></i> -->
      netdata<span style="position: absolute; opacity: 0; width: 0px"
        >system.io</span
      >
    </div>
    <div
      id="easypiechart-b87170f8-0e7a-27fd-9190-81b5377baf35-chart"
      class="netdata-chart netdata-easypiechart-chart"
      style=""
    >
      <span class="easyPieChartLabel" style="font-size: 15px; top: 51px"
        >0.1</span
      ><span
        class="easyPieChartTitle"
        style="font-size: 8px; line-height: 8px; top: 32px"
        >Disk Read</span
      ><span class="easyPieChartUnits" style="font-size: 7px; top: 76px"
        >MiB/s</span
      ><canvas
        height="238"
        width="238"
        style="height: 119px; width: 119px"
      ></canvas>
    </div>
  </div>
  <div
    class="netdata-container-easypiechart"
    style="margin-right: 10px; width: 11%; will-change: transform"
    data-host="http://${Nodes}:19999"
    data-netdata="system.io"
    data-dimensions="out"
    data-chart-library="easypiechart"
    data-title="Disk Write"
    data-width="11%"
    data-before="0"
    data-after="-360"
    data-points="360"
    data-common-units="system.io.mainhead"
    role="application"
  >
    <div
      class="netdata-message icon hidden"
      style="font-size: 12px; padding-top: 51px"
    >
      <svg
        class="svg-inline--fa fa-chart-area fa-w-16"
        aria-hidden="true"
        data-fa-processed=""
        data-prefix="fas"
        data-icon="chart-area"
        role="img"
        xmlns="http://www.w3.org/2000/svg"
        viewBox="0 0 512 512"
      >
        <path
          fill="currentColor"
          d="M500 384c6.6 0 12 5.4 12 12v40c0 6.6-5.4 12-12 12H12c-6.6 0-12-5.4-12-12V76c0-6.6 5.4-12 12-12h40c6.6 0 12 5.4 12 12v308h436zM372.7 159.5L288 216l-85.3-113.7c-5.1-6.8-15.5-6.3-19.9 1L96 248v104h384l-89.9-187.8c-3.2-6.5-11.4-8.7-17.4-4.7z"
        ></path></svg
      ><!-- <i class="fas fa-chart-area"></i> -->
      netdata<span style="position: absolute; opacity: 0; width: 0px"
        >system.io</span
      >
    </div>
    <div
      id="easypiechart-f21f266c-8098-75f2-8ab1-b141b5737dd5-chart"
      class="netdata-chart netdata-easypiechart-chart"
      style=""
    >
      <span class="easyPieChartLabel" style="font-size: 16px; top: 51px"
        >0.2</span
      ><span
        class="easyPieChartTitle"
        style="font-size: 9px; line-height: 9px; top: 30px"
        >Disk Write</span
      ><span class="easyPieChartUnits" style="font-size: 8px; top: 78px"
        >MiB/s</span
      ><canvas
        height="240"
        width="240"
        style="height: 120px; width: 120px"
      ></canvas>
    </div>
  </div>
  <div
    class="netdata-container-gauge"
    style="margin-right: 10px; width: 20%; will-change: transform"
    data-host="http://${Nodes}:19999"
    data-netdata="system.cpu"
    data-chart-library="gauge"
    data-title="CPU"
    data-units="%"
    data-gauge-max-value="100"
    data-width="20%"
    data-after="-360"
    data-points="360"
    data-colors="#22AA99"
    role="application"
  >
    <div
      class="netdata-message icon hidden"
      style="font-size: 21.7px; padding-top: 51.65px"
    >
      <svg
        class="svg-inline--fa fa-chart-area fa-w-16"
        aria-hidden="true"
        data-fa-processed=""
        data-prefix="fas"
        data-icon="chart-area"
        role="img"
        xmlns="http://www.w3.org/2000/svg"
        viewBox="0 0 512 512"
      >
        <path
          fill="currentColor"
          d="M500 384c6.6 0 12 5.4 12 12v40c0 6.6-5.4 12-12 12H12c-6.6 0-12-5.4-12-12V76c0-6.6 5.4-12 12-12h40c6.6 0 12 5.4 12 12v308h436zM372.7 159.5L288 216l-85.3-113.7c-5.1-6.8-15.5-6.3-19.9 1L96 248v104h384l-89.9-187.8c-3.2-6.5-11.4-8.7-17.4-4.7z"
        ></path></svg
      ><!-- <i class="fas fa-chart-area"></i> -->
      netdata<span style="position: absolute; opacity: 0; width: 0px"
        >system.cpu</span
      >
    </div>
    <div
      id="gauge-0d05677f-6e04-6d1b-bbb0-9e3b77f46f8d-chart"
      class="netdata-chart netdata-gauge-chart"
      style=""
    >
      <canvas
        id="gauge-0d05677f-6e04-6d1b-bbb0-9e3b77f46f8d-canvas"
        class="gaugeChart"
        width="434"
        height="260"
        style="width: 217px; height: 130px"
      ></canvas
      ><span class="gaugeChartLabel" style="font-size: 26px; top: 33px"
        >1.0</span
      ><span
        class="gaugeChartTitle"
        style="font-size: 12px; line-height: 12px; top: 0px"
        >CPU</span
      ><span class="gaugeChartUnits" style="font-size: 11px">%</span
      ><span class="gaugeChartMin" style="font-size: 20px">0.0</span
      ><span class="gaugeChartMax" style="font-size: 20px">100.0</span>
    </div>
  </div>
  <div
    class="netdata-container-easypiechart"
    style="margin-right: 10px; width: 11%; will-change: transform"
    data-host="http://${Nodes}:19999"
    data-netdata="system.net"
    data-dimensions="received"
    data-chart-library="easypiechart"
    data-title="Net Inbound"
    data-width="11%"
    data-before="0"
    data-after="-360"
    data-points="360"
    data-common-units="system.net.mainhead"
    role="application"
  >
    <div
      class="netdata-message icon hidden"
      style="font-size: 12px; padding-top: 51px"
    >
      <svg
        class="svg-inline--fa fa-chart-area fa-w-16"
        aria-hidden="true"
        data-fa-processed=""
        data-prefix="fas"
        data-icon="chart-area"
        role="img"
        xmlns="http://www.w3.org/2000/svg"
        viewBox="0 0 512 512"
      >
        <path
          fill="currentColor"
          d="M500 384c6.6 0 12 5.4 12 12v40c0 6.6-5.4 12-12 12H12c-6.6 0-12-5.4-12-12V76c0-6.6 5.4-12 12-12h40c6.6 0 12 5.4 12 12v308h436zM372.7 159.5L288 216l-85.3-113.7c-5.1-6.8-15.5-6.3-19.9 1L96 248v104h384l-89.9-187.8c-3.2-6.5-11.4-8.7-17.4-4.7z"
        ></path></svg
      ><!-- <i class="fas fa-chart-area"></i> -->
      netdata<span style="position: absolute; opacity: 0; width: 0px"
        >system.net</span
      >
    </div>
    <div
      id="easypiechart-952883a8-b560-bee7-715f-49438b44ce39-chart"
      class="netdata-chart netdata-easypiechart-chart"
      style=""
    >
      <span class="easyPieChartLabel" style="font-size: 16px; top: 51px"
        >0.5</span
      ><span
        class="easyPieChartTitle"
        style="font-size: 9px; line-height: 9px; top: 30px"
        >Net Inbound</span
      ><span class="easyPieChartUnits" style="font-size: 8px; top: 78px"
        >megabits/s</span
      ><canvas
        height="240"
        width="240"
        style="height: 120px; width: 120px"
      ></canvas>
    </div>
  </div>
  <div
    class="netdata-container-easypiechart"
    style="margin-right: 10px; width: 11%; will-change: transform"
    data-host="http://${Nodes}:19999"
    data-netdata="system.net"
    data-dimensions="sent"
    data-chart-library="easypiechart"
    data-title="Net Outbound"
    data-width="11%"
    data-before="0"
    data-after="-360"
    data-points="360"
    data-common-units="system.net.mainhead"
    role="application"
  >
    <div
      class="netdata-message icon hidden"
      style="font-size: 11.9px; padding-top: 51.05px"
    >
      <svg
        class="svg-inline--fa fa-chart-area fa-w-16"
        aria-hidden="true"
        data-fa-processed=""
        data-prefix="fas"
        data-icon="chart-area"
        role="img"
        xmlns="http://www.w3.org/2000/svg"
        viewBox="0 0 512 512"
      >
        <path
          fill="currentColor"
          d="M500 384c6.6 0 12 5.4 12 12v40c0 6.6-5.4 12-12 12H12c-6.6 0-12-5.4-12-12V76c0-6.6 5.4-12 12-12h40c6.6 0 12 5.4 12 12v308h436zM372.7 159.5L288 216l-85.3-113.7c-5.1-6.8-15.5-6.3-19.9 1L96 248v104h384l-89.9-187.8c-3.2-6.5-11.4-8.7-17.4-4.7z"
        ></path></svg
      ><!-- <i class="fas fa-chart-area"></i> -->
      netdata<span style="position: absolute; opacity: 0; width: 0px"
        >system.net</span
      >
    </div>
    <div
      id="easypiechart-c7b79f11-6959-1335-9716-a6a05da05e56-chart"
      class="netdata-chart netdata-easypiechart-chart"
      style=""
    >
      <span class="easyPieChartLabel" style="font-size: 15px; top: 51px"
        >0.5</span
      ><span
        class="easyPieChartTitle"
        style="font-size: 8px; line-height: 8px; top: 32px"
        >Net Outbound</span
      ><span class="easyPieChartUnits" style="font-size: 7px; top: 76px"
        >megabits/s</span
      ><canvas
        height="238"
        width="238"
        style="height: 119px; width: 119px"
      ></canvas>
    </div>
  </div>
  <div
    class="netdata-container-easypiechart"
    style="margin-right: 10px; width: 9%; will-change: transform"
    data-host="http://${Nodes}:19999"
    data-netdata="system.ram"
    data-dimensions="used|buffers|active|wired"
    data-append-options="percentage"
    data-chart-library="easypiechart"
    data-title="Used RAM"
    data-units="%"
    data-easypiechart-max-value="100"
    data-width="9%"
    data-after="-360"
    data-points="360"
    data-colors="#EE9911"
    role="application"
  >
    <div
      class="netdata-message icon hidden"
      style="font-size: 9.8px; padding-top: 41.1px"
    >
      <svg
        class="svg-inline--fa fa-chart-area fa-w-16"
        aria-hidden="true"
        data-fa-processed=""
        data-prefix="fas"
        data-icon="chart-area"
        role="img"
        xmlns="http://www.w3.org/2000/svg"
        viewBox="0 0 512 512"
      >
        <path
          fill="currentColor"
          d="M500 384c6.6 0 12 5.4 12 12v40c0 6.6-5.4 12-12 12H12c-6.6 0-12-5.4-12-12V76c0-6.6 5.4-12 12-12h40c6.6 0 12 5.4 12 12v308h436zM372.7 159.5L288 216l-85.3-113.7c-5.1-6.8-15.5-6.3-19.9 1L96 248v104h384l-89.9-187.8c-3.2-6.5-11.4-8.7-17.4-4.7z"
        ></path></svg
      ><!-- <i class="fas fa-chart-area"></i> -->
      netdata<span style="position: absolute; opacity: 0; width: 0px"
        >system.ram</span
      >
    </div>
    <div
      id="easypiechart-34ec531c-54bb-ad66-5d01-484e7a26f2fb-chart"
      class="netdata-chart netdata-easypiechart-chart"
      style=""
    >
      <span class="easyPieChartLabel" style="font-size: 13px; top: 41px"
        >58.7</span
      ><span
        class="easyPieChartTitle"
        style="font-size: 7px; line-height: 7px; top: 25px"
        >Used RAM</span
      ><span class="easyPieChartUnits" style="font-size: 6px; top: 62px">%</span
      ><canvas
        height="196"
        width="196"
        style="height: 98px; width: 98px"
      ></canvas>
    </div>
  </div>
</div>

and repeat the panel based on the nodes dashboard variable.

The panels don’t always render nicely the first time, and sometimes require a refresh - I’m still figuring this out.

system · May 20, 2022, 8:53am

This topic was automatically closed after 365 days. New replies are no longer allowed.

Topic		Replies	Views
Want to create filters for my dashboard panel Grafana templating	1	5121	November 6, 2017
Labels in graphs Grafana	1	4481	December 31, 2018
Automatically generating new dashboards for different data sources Elasticsearch templating	0	343	November 28, 2019
How do I create desktop variables for "yourhostname" and "host"	1	344	January 13, 2021
Dynamic panels based on variable Grafana	4	3743	July 6, 2018

Help me do something crazy cool but also maybe crazy (?) with templating

Related topics