Hi!
I need some suggestions to config log collector agents.
I have a docker swarm cluster with two sort of services - infrastructure and application services.
There are Loki and Promtail services as a part of infrastructure. Responsibility of these services is to collect logs of any docker services deployed on swarm.
By design infrastructure services have to be independent on any application services, deployed on top of them.
Docker delivers the logs to journald and Promtail container is configured to scrape this logs and pass it to Loki.
Problem
I need find some way to configure log parsing as a part of pipeline stage individually for each application service. Because of design decision - infrastructure layer does not “know” how to parse particular application logs. Application log structure have to be described on application layer, not on infrastructure layer.
I need to find some way to config Promtail (or Grafana Agent) within application docker compose file or application container itself.
In other words I need to delegate agent pipeline stage configuration to application service layer.
So the only way I can resolve this task - is to have multiple agents - one agent per application service.
If it is the only way to do so, how to properly config multiple agent containers on same host machine to scrape the same host journald?
Is it right decision to mount - /var/log/journal:/var/log/journal:ro to multiple containers and filter it before send logs to Loki?
Is there any performance impact or concurrency issues to be expected?