Hello,
Looking to know if it’s possible without building a custom interface with the loki api to ship logs out of loki to external systems ?
I’m looking at options for shipping existing logs that have already been delivered to loki and pulling them into Kafka / kinesis stream so other services can process the log data.
I don’t think there is a built-in method to forward logs out of Loki, you’ll have to build some sort of API to do that.
That said, and I could be talking nonsense here since I don’t have knowledge into your setup or use case, but if you are looking to process events out of streaming platform such as Kafka usually you’d expect it to be pretty real time. If you have to rely on logs being shipped to Loki first wouldn’t that be added point of failure as well as potential latency? Why not have the source (log pipelines, since promtail doesn’t do kafka that I know, perhaps use something like fluentd) output two log streams, one to Loki and one to Kafka? Or, you could have everything go to Kafka instead, and add an extractor for things you want to keep in Loki (this gives you added benefit of using Kafka as your buffer, though that’s probably not really necessary unless you are doing terabytes of logs, even then Loki might be able to handle it).
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.