Connection refused with Loki and Promtail setup

Grafana Loki
1

I’m trying to configure Loki + Promtail in my Debian Buster.

Running Loki:

root@localhost:~# /usr/local/bin/loki -config.file /etc/loki/loki-local-config.yaml
    level=info ts=2020-12-22T04:33:04.320604725Z caller=main.go:128 msg="Starting Loki" version="(version=2.0.0, branch=HEAD, revision=6978ee5d)"
    level=info ts=2020-12-22T04:33:04.321407127Z caller=server.go:225 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
    level=info ts=2020-12-22T04:33:04.322338311Z caller=shipper_index_client.go:97 msg="starting boltdb shipper in 0 mode"
    level=info ts=2020-12-22T04:33:04.322767112Z caller=worker.go:65 msg="no address specified, not starting worker"
    level=info ts=2020-12-22T04:33:04.323359953Z caller=mapper.go:38 msg="cleaning up mapped rules directory" path=/tmp/loki/rules-temp
    level=error ts=2020-12-22T04:33:04.323498655Z caller=mapper.go:42 msg="unable to read rules directory" path=/tmp/loki/rules-temp err="open /tmp/loki/rules-temp: no such file or directory"
    level=info ts=2020-12-22T04:33:04.324250337Z caller=table_manager.go:171 msg="uploading tables"
    level=info ts=2020-12-22T04:33:04.325102699Z caller=module_service.go:58 msg=initialising module=memberlist-kv
    level=info ts=2020-12-22T04:33:04.325165469Z caller=module_service.go:58 msg=initialising module=server
    level=info ts=2020-12-22T04:33:04.325328069Z caller=module_service.go:58 msg=initialising module=store
    level=info ts=2020-12-22T04:33:04.32541861Z caller=module_service.go:58 msg=initialising module=ingester
    level=info ts=2020-12-22T04:33:04.3254849Z caller=lifecycler.go:518 msg="not loading tokens from file, tokens file path is empty"
    level=info ts=2020-12-22T04:33:04.325558011Z caller=lifecycler.go:547 msg="instance not found in ring, adding with no tokens" ring=ingester
    level=info ts=2020-12-22T04:33:04.325694941Z caller=lifecycler.go:394 msg="auto-joining cluster after timeout" ring=ingester
    level=info ts=2020-12-22T04:33:04.325889032Z caller=module_service.go:58 msg=initialising module=table-manager
    level=info ts=2020-12-22T04:33:04.326499654Z caller=table_manager.go:324 msg="synching tables" expected_tables=60
    level=info ts=2020-12-22T04:33:04.326569524Z caller=table_manager.go:476 msg="creating table" table=index_18565
    ...
    level=info ts=2020-12-22T04:33:04.327176795Z caller=table_manager.go:476 msg="creating table" table=index_18603
    level=info ts=2020-12-22T04:33:04.327222806Z caller=module_service.go:58 msg=initialising module=ring
    level=info ts=2020-12-22T04:33:04.327336696Z caller=module_service.go:58 msg=initialising module=compactor
    level=info ts=2020-12-22T04:33:04.327419476Z caller=module_service.go:58 msg=initialising module=ingester-querier
    level=info ts=2020-12-22T04:33:04.327450036Z caller=module_service.go:58 msg=initialising module=distributor
    level=info ts=2020-12-22T04:33:04.327506307Z caller=module_service.go:58 msg=initialising module=ruler
    level=info ts=2020-12-22T04:33:04.327534387Z caller=ruler.go:400 msg="ruler up and running"
    level=info ts=2020-12-22T04:33:04.327616038Z caller=loki.go:227 msg="Loki started"

Running Promtail:

root@localhost:~# ./promtail-linux-amd64 -config.file /etc/promtail/config-promtail.yml 
level=info ts=2020-12-22T04:21:31.582053518Z caller=server.go:225 http=[::]:9080 grpc=[::]:38035 msg="server listening on addresses"
level=info ts=2020-12-22T04:21:31.582712947Z caller=main.go:108 msg="Starting Promtail" version="(version=2.0.0, branch=HEAD, revision=6978ee5d)"
level=warn ts=2020-12-22T04:21:32.589601787Z caller=client.go:288 component=client host=127.0.0.1:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://127.0.0.1:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"

With Loki configuration from

https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml

And promtail config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit

What could be wrong with my configuration?

2

I don’t really see anything wrong with your config, I would be suspicious of a firewall rules or iptables rules preventing traffic on localhost.

Can you curl Loki directly?

❯ curl http://127.0.0.1:3100/ready
ready
4

@ewelch hi,

root@localhost:~# curl http://127.0.0.1:3100/ready
ready

And I downloaded the latest promtail then run it again now getting this error:

root@localhost:~# sudo ./promtail -config.file ./config-promtail.yml
Unable to parse config: ./config-promtail.yml: yaml: unmarshal errors:
  line 1: field http_listen_port not found in type main.Config
  line 2: field grpc_listen_port not found in type main.Config
5

I managed to get Loki + promtail working, not sure how exactly, but I just downloaded the release package again, then run it the same way.

As for the config problem, there’s a missing “server:” in the first line the is causing it.

6

I faced similar errors until I realized that I was running promtail before loki. Once I ran Loki before promtail it worked.