Hi Everyone,
I would like to setup scalable and highly available loki cluster with multiple nodes establishing a ring and adding a memberlist config but only problem i face here is i had to add the node ip in all the loki config for a new loki instance,
Is there any easy way to dynamically pass or any other way we can implement instead of passing ip address of the loki instance?
I’m thinking to add a autoscaling group with target groups and scale-in and scale-out based on demand of resources.
i’m referring standalone ec2 deployment and already setup monolothic loki deployment mode with 2 nodes forming a cluster but want to add a autoscaling feature for this deployment since scaling is dynamic how can we add a new ip address each time instance is scaled-out and removing ip address when scaled in in asg
If you are doing monolithic deployment on EC2, you can create a DNS record with multiple values pointing to both of your nodes, and use that for your ring membership.
Hi @tonyswumac
I have setup the similar way along with ALB pointing to 2 nodes as a part of target group.
I feel DNS with multiple values are always cached so enhanced the setup with ALB with disabling sticky sessions
Was thinking to enhance it more with auto scaling setup if there is no easy way I’m thinking to write a script to get the ip and update it in each loki instance when there is a scaling threshold matches
You don’t want to use ALB for ring membership. Load balancer is an endpoint itself. You hit the load balancer, and the load balancer decides which backend to forward your request to, that’s not what you want.
For a ring membership, every member has to know every other member. In Docker environment this would be done via service discovery. In your case DNS record with multiple values. To be clear. ring membership isn’t doing DNS round robin, it’s parsing the values returned from the DNS record so it knows what members are within the ring.
Edit: If you think about it, it’s actually the same thing as hard coding your two instance IPs in the configuration, except in this case you are using a DNS record so it’s dynamic and you can change your instance without worrying about IP changing (as long as you update the DNS record).
