Loki microservice with memberlist ring store

Grafana Configuration
1

Has anyone got an example configuration for running an ingester, distributor, and querier as microservices on the same server using memberlist as the ring store?

Using consul I have this working - well, logs get to S3, and I can query them. However, it would be good to have a zero-dependencies setup, so we’d like to use memberlist for ring storage. With consul, I can browse to /ring, and see the cortex ring status, with 3 members (I’m running the a set of ingester/distributor/querier on each of 3 VMs).

If I try to use memberlist, I constantly get errors about the ring being empty, and the /ring url shows no members.

In the ingester.yaml, I have this:
ring:
kvstore:
store: memberlist
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
max_transfer_retries: 0

memberlist:
node_name: ${HOSTNAME}
bind_port: 7946
join_members:
- loki1:7946
- loki2:7946
- loki3:7946
abort_if_cluster_join_fails: false

which is the same on all 3 servers, and for the distributor.yaml, I have the same stanza. All 3 components run on each of 3 servers.

Only the ingester process shows port 7946 - the (bind port for the ring) in the output of ss -ntlp, irrespective of changing the bind address in the other component config files, which is actually what I expected, but I also expected that the other components would join the ring by talking to that port. I’m very confused, and suspect that a working example using memnerlist - even for a single component to VM, but with more than one vm.

By running 2 ingesters on one host (with different ports), I have come to the conclusion that the memberlist ring does not update the /ring URL - would that be correct.

Thanks

2

To summarise: should I have a memberlist: block in every component, or just the ingester component?
Either way, I see this error:
evel=error ts=2020-09-11T15:56:32.447558222Z caller=pool.go:161 msg=“error removing stale clients” err=“empty ring”
in the distributor log.

Whereas the ingesters seem to talk to each other - this is from an ingester log:

ts=2020-09-11T16:00:31.134947018Z caller=memberlist_logger.go:74 level=debug msg=“Initiating push/pull sync with: 192.168.50.39:7946”
ts=2020-09-11T16:00:31.135230033Z caller=memberlist_logger.go:74 level=debug msg=“Stream connection from=192.168.50.39:44040”
ts=2020-09-11T16:00:31.139790956Z caller=memberlist_logger.go:74 level=debug msg=“Initiating push/pull sync with: 192.168.50.41:7946”
ts=2020-09-11T16:00:31.145082204Z caller=memberlist_logger.go:74 level=debug msg=“Initiating push/pull sync with: 192.168.50.42:7946”
level=info ts=2020-09-11T16:00:31.147945381Z caller=memberlist_client.go:460 msg=“joined memberlist cluster” reached_nodes=3

Thanks

3

A long shot but… have you figured whats up?

I have ingester and distributor in a GKE cluster but the ring is empty even though both advertise ports - both on different nodes.