Multiple Loki different inctances send to same bucket

[christoforosd]

I have three different instances of Loki, they are in different region but i write them to the same Bucket.

The issue is that from the main Loki - Grafana it take very long time to show the logs from the other two loki instances (usually 50m-1h).

I try to disable compactor but loki failed. I didn’t join the instances together with memberlist. Not sure if i need to join to work this.

Any ideas please ?

compactor:
  working_directory: /data/loki/boltdb-shipper-compactor
  shared_store: aws

table_manager:
  retention_deletes_enabled: true
  retention_period: 720h

Thanks

[tonyswumac]

Loki querier queries the writer instances within configured query_ingesters_within. If your querier can’t reach the remote writers then that would explain it.

You could try to lower it, but a better option would be to re-design your architecture. You should keep Loki cluster container within one location, then from your remote accounts setup promtail or fluend forwarders to forward logs to your Loki cluster. The connectivity can be facilitated with VPC endpoint or transit gateway.

[christoforosd]

I only query the main instance. But if i keep Loki Cluster container to one location then the price it will be a lot more. And the SG it will be a nightmare to manage them.

Loki don’t support this architecture?
Write from multiple Loki to same bucket, if i use DynamoDB it should be ok?

[christoforosd]

I change config query_ingesters_within to all three instances to 10m, but still have issue maybe 30m. If i change from boltdb-shipper to DynamoDB the issue it will be fix?

I didn’t understand why is happening that. Is because writer instances with config query_ingesters_within it take too much time to ingest ?

[tonyswumac]

There is a limit on how often writers can write to chunk storage, it’s controlled either by size, idle period, or max wait period. Typically you’d configure query_ingesters_within within the max wait period.

I don’t know why the cost would be a lot more, you’d be running the same number of hosts (maybe even less), and I am not sure why it would be a nightmare to manage.

[christoforosd]

If i have promtail to send all logs to one region then i will paid for all the traffic pass from transit gateway.

[tonyswumac]

VPC endpoint service might be an alternative then.

It’s a trade-off, if you think about it. You either accept that there will always be some logs living on the ingester for a period of time before it’s sent to chunk storage, or you try to justify the cost.

[christoforosd]

Thank you a lot for your help

Stupid question if i index storage from boltdb-shipper to DynamoDB and all three nodes write to DynamoDB then this issue it will be better or it will be the same.

I was thinking that the writers, write to chunk storage instantly and just the search it take time to search the storage - index ?

[slappeyibkr]

@tonyswumac how do I configure the querier to query the ingesters? How does the querier learn about the ingesters? Over cluster gossip? I can’t figure this part out. I’m able to query logs from minio but not from the ingesters. I’ve tried everything I can think of and nothing has worked for me. My query_ingesters_within is default of 3h.

[tonyswumac]

@christoforosd

Thank you a lot for your help

Stupid question if i index storage from boltdb-shipper to DynamoDB and all three nodes write to DynamoDB then this issue it will be better or it will be the same.

I was thinking that the writers, write to chunk storage instantly and just the search it take time to search the storage - index ?

It probably wouldn’t make any difference, your queriers still need to be able to connect to ingesters for logs that haven’t been written to chunk storage.

[tonyswumac]

@slappeyibkr

@tonyswumac how do I configure the querier to query the ingesters? How does the querier learn about the ingesters? Over cluster gossip? I can’t figure this part out. I’m able to query logs from minio but not from the ingesters. I’ve tried everything I can think of and nothing has worked for me. My query_ingesters_within is default of 3h.

Please see my reply to another topic here: https://localhost:3000/t/loki-2-4-1-empty-ring-code-500-error-for-get-loki-api-v1-labels-api-on-aws-ecs/84964/10

It doesn’t directly apply to you (unless you are using ECS as well), but I did explain how service discovery works and how memberlist ring is configured there.

[slappeyibkr]

@tonyswumac

n’t directly apply to you (unless you are using ECS as well), but I did explain how service discovery works and how memberlist ring is configured there.

When you say “it doesn’t propagate the port which means you have to use native port (meaning no bridge) for your writers.”, what writer port are you referring to? gRPC port? or gossip port? I’m assumnig gRPC. I have all memberlist members communicating successfully over gossip. We are using podman with slirp4netns network mode and port translation. For gossip to work, we had to configure advertise_addr and advertise_port. I just can’t seem to figure out how to get the querier to query the ingesters.

[tonyswumac]

@slappeyibkr

When you say “it doesn’t propagate the port which means you have to use native port (meaning no bridge) for your writers.”, what writer port are you referring to? gRPC port? or gossip port?

Just in general, you’d want all open ports from writers to be directly accessible within the cluster. I see you created another topic, perhaps we should move this conversation there (just tag me) instead of hijacking someone else’s topic.

Also please include your loki configuration (particularly the ring memberlist part), and test telnet from your querier to writer and see if they work or not.