Promtail - log file does not match glob - but it does?

Grafana Loki
1

Platform: linux/amd64
Promtail: 2.3.0

I’m not seeing what I’m doing wrong here.

I have a process only writing once a day to a log file. It can happen that the log file doesn’t exist at the time that promtail started. When the log file appears, promtail does not want to tail it claiming that “new file does not match glob”

Relevant config:

scrape_configs:
- job_name: prometheus
  static_configs:
  - labels:
      job: prometheus_exporters
      __path__: /opt/prometheus/log/*exporter*log
  - labels:
      job: prometheus
      __path__: /opt/prometheus/log/prometheus.log
  - labels:
      job: prometheus_backup
      __path__: /opt/prometheus/log/prometheus_backup.log

When promtail starts with log file absent

level=debug ts=2021-09-09T11:00:34.865406137Z caller=filetarget.go:197 msg="no files matched requested path, nothing will be tailed" path=/opt/prometheus/log/prometheus_backup.log
level=info ts=2021-09-09T11:00:34.865461662Z caller=filetargetmanager.go:271 msg="Removing target" key="{job=\"prometheus_backup\"}"
level=info ts=2021-09-09T11:00:34.865584982Z caller=filetarget.go:151 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/opt/prometheus/log/prometheus_backup.log

When I force creation of the log file, I see this appearing:

level=debug ts=2021-09-09T11:12:00.205331105Z caller=filetarget.go:168 msg="new file does not match glob" filename=/opt/prometheus/log/prometheus_backup.log
level=debug ts=2021-09-09T11:12:00.206098947Z caller=filetarget.go:168 msg="new file does not match glob" filename=/opt/prometheus/log/prometheus_backup.log

Checking by copy pasting path from log message

-rw-rw-r-- 1 dwpmetrc dwpmetrc 0 Sep 9 13:15 /opt/prometheus/log/prometheus_backup.log

What am I overlooking?

2

Hhmm, interesting.

Your scrape configs don’t appear right. You’re missing static_configs; it should look like this:

scrape_configs:
- job_name: prometheus
  static_configs:
    - targets:
        - localhost
      labels:
        job: prometheus_backup
        __path__: /opt/prometheus/log/prometheus_backup.log

Relevant docs: Configuration | Grafana Labs

3

I made an error when copy-pasting - original post corrected

4

OK - so the debug message just compares any new file against the existing file targets. After adding my own debug message in loki/clients/pkg/promtail/targets/file/filetarget.go and adding a dodo.log to my directory:

level=debug ts=2021-09-15T12:35:33.819640489Z caller=filetarget.go:167 msg=WHY error=null match=false target.path=/opt/prometheus/log/prometheus_backup.log filename=/opt/prometheus/log/dodo.log
level=debug ts=2021-09-15T12:35:33.819857511Z caller=filetarget.go:169 msg="new file does not match glob" filename=/opt/prometheus/log/dodo.log
level=debug ts=2021-09-15T12:35:33.819875777Z caller=filetarget.go:167 msg=WHY error=null match=false target.path=/opt/prometheus/log/prometheus.log filename=/opt/prometheus/log/dodo.log
level=debug ts=2021-09-15T12:35:33.819939117Z caller=filetarget.go:169 msg="new file does not match glob" filename=/opt/prometheus/log/dodo.log

However it seems that prometheus does not check if a filetarget which wasn’t present at its launch, appears later. It just removes that filetarget at launch

level=debug ts=2021-09-15T12:38:31.88237282Z caller=filetarget.go:198 msg="no files matched requested path, nothing will be tailed" path=/opt/prometheus/log/prometheus_backup.log
level=info ts=2021-09-15T12:38:31.882478211Z caller=filetargetmanager.go:271 msg="Removing target" key="{job=\"prometheus_backup\"}"

Is there any way to keep filetargets not existing at launch and make promtail tail them when they do appear?

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.