I’m developing a Docker Swarm application and I have a problem with Loki.
Sometimes I get this error on the logs:
level=error caller=scheduler_processor.go:87 msg=“error contacting scheduler” err=“rpc error: code = Unavailable desc = connection error: desc= "transport: Error while dialing dial tcp 10.0.0.47:9095 i/o timeout"” addr=10.0.0.47:9095
I said that sometimes I get this error because randomly after some new deploys it runs like a charm.
The strange thing is also that the port 9095 ins’t used by anyone inside my swarm so I really don’t know why it says that cannot dial that port.
This is how I have created the service inside my docker-compose.yml:
...
mon_loki:
image: grafana/loki:2.5.0
hostname: mon_loki
restart: always
ports:
- 3100:3100
command: -config.file=/etc/loki/local-config.yaml
volumes:
- /data/loki/config.yaml:/etc/loki/config.yaml
- /data/loki:/data/loki
deploy:
placement:
constraints:
- node.role == worker
depends_on:
- mon_node-exporter
- mon_cadvisor
networks:
- docker
...
This is loki’s config.yaml
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
schema_config:
configs:
- from: 2020-05-15
store: boltdb
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h
storage_config:
boltdb:
directory: /tmp/loki/index
filesystem:
directory: /tmp/loki/chunks
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h