One type of log (MySQL) don't ship to loki

Grafana Loki
1

Describe the bug
One type of log don’t shipped from all servers. I have about 40 servers with Promtail which send logs to loki server.
Almost all servers have same type of logs. That is syslog, authlog, weblog, mysql, php …

All logs from server go well except MySQL logs.

Promtail config:

scrape_configs:
  - job_name: syslog
    static_configs:
    - labels:
        __path__: /var/log/syslog
        host: myhostname
        job: syslog
      targets:
      - localhost
  - job_name: mysql
      static_configs:
      - labels:
          __path__: /var/log/mysql/*.log
          host: myhostname
          job: mysql
        targets:
        - localhost
  - job_name: nginx
    static_configs:
    - labels:
        __path__: /var/log/nginx/*log
        host: myhostname
        job: nginx
      targets:
      - localhost
  ....

For every job, I’m using different job_name and different label job.

All jobs and logs come to loki and I can see it except MySQL logs.

Logs and path exists:

ls -la /var/log/mysql
-rw-r-----  1 mysql adm      24869 Jun  2  2020 error.log
-rw-r-----  1 mysql mysql  1109678 Feb 23 10:18 mysql-error.log
-rwxr-xr-x  1 mysql mysql    20535 Feb 11 04:00 mysql-slow.log

Promtail works under root user and permission is not problem.

root     30597  0.3  1.5 1977696 61440 ?       Ssl  12:02   0:04 /usr/local/bin/promtail -config.file=/etc/promtail/promtail.yml -log.level=info

From service status everything looks good

promtail[5617]: ts=2022-02-23T11:26:36.504729542Z caller=log.go:168 level=info msg="Seeked /var/log/mysql/mysql-error.log - &{Offset:1109678 Whence:0}"
promtail[5617]: level=info ts=2022-02-23T11:26:36.504743296Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/mysql/mysql-error.log
promtail[5617]: ts=2022-02-23T11:26:36.504766577Z caller=log.go:168 level=info msg="Seeked /var/log/mysql/mysql-slow.log - &{Offset:20535 Whence:0}"
promtail[5617]: level=info ts=2022-02-23T11:26:36.504778003Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/mysql/mysql-slow.log
promtail[5617]: level=info ts=2022-02-23T11:26:36.504785215Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/mysql/error.log

To Reproduce

Expected behavior
Expect to see MySQL job and logs in loki.

Environment:

  • Infrastructure: Virtual Machine (Ubuntu 20.04)
  • Deployment tool: [e.g., helm, jsonnet]

Screenshots, Promtail config, or terminal output

Promtail version:

 promtail --version
promtail, version 2.4.1 (branch: HEAD, revision: f61a4d261)
  build user:       root@9cbe2f86250f
  build date:       2021-11-08T13:05:10Z
  go version:       go1.16.2
  platform:         linux/amd64

Loki version:

loki, version 2.4.1 (branch: HEAD, revision: f61a4d261)
  build user:       root@9cbe2f86250f
  build date:       2021-11-08T13:05:10Z
  go version:       go1.16.2
  platform:         linux/amd64
1 Like
2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.