Fetch logs from an S3 bucket to Loki?

Grafana Loki
1

Does anyone have experience with fetching logs from an S3 bucket to Loki?

My bucket is S3-compatible and is not an AWS service [^1], so I can’t use lambda-promtail to solve the problem.

Another idea that comes to my mind is to use Logstash with S3 input plugin for feeding the logs to Loki instead. But I don’t really want to provision an additional Logstash. Is there any way to solve the problem with Promtail?

This is a duplicated question. The thread is locked though, and I don’t particularly like the process of mounting an S3 bucket as a virtual filesystem.

[^1] For anyone curious, it’s this OBS (Object Storage Service) by a cloud provider in Germany.

2

If you are set to using promtail then S3FS is probably your best bet. I haven’t used it personally but I’ve heard it’s reasonably stable.

Another solution that we do sometimes is to just use a simple script to pull from S3 bucket. Obviously you’ll need to do some coding here, and depending on how you structure your bucket (or don’t structure) how you approach the scripting would be different.

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.