I’m using Prometheus to scrape grok_exporter metrics (from log files). I have the Prometheus data source Scrape Interval set to 5s.
When I tail the source logfile, I see the correct data:
May 2 20:16:03 172.16.50.150 /var/log/pald.log[839] MALONE UPS 2: Output Load = 20
May 2 20:16:12 172.16.50.150 /var/log/pald.log[839] MALONE UPS 2: Input Voltage = 118.8
May 2 20:16:23 172.16.50.150 /var/log/pald.log[821] MALONE UPS 1: Input Voltage = 119.3
May 2 20:16:32 172.16.50.150 /var/log/pald.log[821] MALONE UPS 1: Output Current = 5.00
May 2 20:16:33 172.16.50.150 /var/log/pald.log[821] MALONE UPS 1: Output Current = 4.00
May 2 20:16:42 172.16.50.150 /var/log/pald.log[821] MALONE UPS 1: Input Voltage = 119.6
May 2 20:16:49 172.16.50.150 /var/log/pald.log[839] MALONE UPS 2: Input Voltage = 119.3
May 2 20:16:59 172.16.50.150 /var/log/pald.log[839] MALONE UPS 2: Output Load = 19
Looking at the /metrics endpoint, looks right to me?
# HELP ups ups gauge
# TYPE ups gauge
ups{id="1",location="MALONE",measurement="Battery Voltage"} 54.6
ups{id="1",location="MALONE",measurement="Input Frequency"} 59.7
ups{id="1",location="MALONE",measurement="Input Voltage"} 119.8
ups{id="1",location="MALONE",measurement="Output Current"} 4
ups{id="1",location="MALONE",measurement="Output Load"} 29
ups{id="2",location="MALONE",measurement="Input Frequency"} 59.7
ups{id="2",location="MALONE",measurement="Input Voltage"} 120.1
ups{id="2",location="MALONE",measurement="Output Current"} 3
ups{id="2",location="MALONE",measurement="Output Load"} 19
In Prometheus, if I look at the same timestamps, the data is there, but it’s slightly incorrect. What’s going on here?
Element Value
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Battery Voltage"} 54.6
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Input Frequency"} 59.7
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Input Voltage"} 119.8
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Output Current"} 4
ups{id="1",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Output Load"} 29
ups{id="2",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Input Frequency"} 59.7
ups{id="2",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Input Voltage"} 120.1
ups{id="2",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Output Current"} 3
ups{id="2",instance="172.16.50.150:9144",job="grok",location="MALONE",measurement="Output Load"} 19
But here’s the kicker… the values never change.
the grok_exporter config.yml is set to poll the logfile every 5 seconds. One clue is that if I set readall to false, I get no data at all. I can delete the logfile, it is re-created automatically, and data starts to flow again. But, its the same data over and over lol.
global:
config_version: 2
input:
type: file
path: /var/log/upsboth.log
fail_on_missing_logfile: false
poll_interval: 5s
readall: true
grok:
patterns_dir: ./logstash-patterns-core/patterns
metrics:
- type: gauge
name: ups
help: ups gauge
match: '%{MONTH:month}%{SPACE}%{MONTHDAY:day}%{SPACE}%{TIME:time}%{SPACE}%{USER}%{SPACE}%{PATH}%{NOTSPACE}%{NUMBER}%{NOTSPACE}%{SPACE}%{NOTSPACE:location}%{SPACE}%{NOTSPACE:asset}%{SPACE}%{INT:id}%{NOTSPACE}%{SPACE}(?<measurement>%{WORD}%{SPACE}%{WORD})%{SPACE}%{NOTSPACE}%{SPACE}%{NUMBER:val}'
value: '{{.val}}'
cumulative: false
labels:
location: '{{.location}}'
id: '{{.id}}'
measurement: '{{.measurement}}'
host: localhost
port: 9144