Prometheus metrics not updating

Grafana
1

I’m using Prometheus to scrape grok_exporter metrics (from log files). I have the Prometheus data source Scrape Interval set to 5s.

When I tail the source logfile, I see the correct data:

May  2 20:16:03 172.16.50.150 /var/log/pald.log[839] MALONE UPS 2: Output Load = 20
May  2 20:16:12 172.16.50.150 /var/log/pald.log[839] MALONE UPS 2: Input Voltage = 118.8
May  2 20:16:23 172.16.50.150 /var/log/pald.log[821] MALONE UPS 1: Input Voltage = 119.3
May  2 20:16:32 172.16.50.150 /var/log/pald.log[821] MALONE UPS 1: Output Current = 5.00
May  2 20:16:33 172.16.50.150 /var/log/pald.log[821] MALONE UPS 1: Output Current = 4.00
May  2 20:16:42 172.16.50.150 /var/log/pald.log[821] MALONE UPS 1: Input Voltage = 119.6
May  2 20:16:49 172.16.50.150 /var/log/pald.log[839] MALONE UPS 2: Input Voltage = 119.3
May  2 20:16:59 172.16.50.150 /var/log/pald.log[839] MALONE UPS 2: Output Load = 19

Looking at the /metrics endpoint, looks right to me?

# HELP ups ups gauge
# TYPE ups gauge
ups{id="1",location="MALONE",measurement="Battery Voltage"} 54.6
ups{id="1",location="MALONE",measurement="Input Frequency"} 59.7
ups{id="1",location="MALONE",measurement="Input Voltage"} 119.8
ups{id="1",location="MALONE",measurement="Output Current"} 4
ups{id="1",location="MALONE",measurement="Output Load"} 29
ups{id="2",location="MALONE",measurement="Input Frequency"} 59.7
ups{id="2",location="MALONE",measurement="Input Voltage"} 120.1
ups{id="2",location="MALONE",measurement="Output Current"} 3
ups{id="2",location="MALONE",measurement="Output Load"} 19

In Prometheus, if I look at the same timestamps, the data is there, but it’s slightly incorrect. What’s going on here?


Element	Value
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Battery Voltage"}	54.6
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Input Frequency"}	59.7
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Input Voltage"}	119.8
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Output Current"}	4
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Output Load"}	29
ups{id="2",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Input Frequency"}	59.7
ups{id="2",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Input Voltage"}	120.1
ups{id="2",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Output Current"}	3
ups{id="2",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Output Load"}	19

But here’s the kicker… the values never change.

image
image1135×460 66.6 KB

the grok_exporter config.yml is set to poll the logfile every 5 seconds. One clue is that if I set readall to false, I get no data at all. I can delete the logfile, it is re-created automatically, and data starts to flow again. But, its the same data over and over lol.

global:
    config_version: 2
input:
    type: file
    path: /var/log/upsboth.log
    fail_on_missing_logfile: false
    poll_interval: 5s
    readall: true
grok:
    patterns_dir: ./logstash-patterns-core/patterns
metrics:
    - type: gauge
      name: ups
      help: ups gauge
      match: '%{MONTH:month}%{SPACE}%{MONTHDAY:day}%{SPACE}%{TIME:time}%{SPACE}%{USER}%{SPACE}%{PATH}%{NOTSPACE}%{NUMBER}%{NOTSPACE}%{SPACE}%{NOTSPACE:location}%{SPACE}%{NOTSPACE:asset}%{SPACE}%{INT:id}%{NOTSPACE}%{SPACE}(?<measurement>%{WORD}%{SPACE}%{WORD})%{SPACE}%{NOTSPACE}%{SPACE}%{NUMBER:val}'
      value: '{{.val}}'
      cumulative: false
      labels:
          location: '{{.location}}'
          id: '{{.id}}'
          measurement: '{{.measurement}}'
host: localhost
port: 9144
2

Upon further diagnosis, the data updates only when the grok_exporter service is restarted, and then not again after that.

Here’s what happens when I restart the grok_exporter container:

image
image743×246 20.6 KB

I can go into the container and tail the log file, the data is changing, yet prometheus and grafana just keep pushing the same values regardless. I’m having a hard time comprehending where that data is coming from.

3

Solved. :upside_down_face: :checkered_flag:

The issue was in grok_exporter config. I used a version 3 flag in a version 2 configuration file. In the config.yml, I had to change this:

input:
    type: file
    path: /var/log/upsboth.log
    fail_on_missing_logfile: false
    poll_interval: 5s
    readall: true

to this:

input:
    type: file
    path: /var/log/upsboth.log
    fail_on_missing_logfile: false
    poll_interval_seconds: 5
    readall: true

and now I have pretty metrics that update!

image

1 Like