Hi all,
I’ve deploy a setup of loki/promtail on a cloud VPS and am testing displaying inbound logs from 4 other systems with grafana. I’m displaying a graph, counting log entries per interval of the files in /var/log/* (no subdirs) of one promtail client. Loki is running on file based data store.
For one system/graph display counting logs of the last 24 hours, it saturates 8 cores and 8GB mem (of 16gb) up to 30 seconds easily. Which will probably get worse when I start adding another dozen plus servers. I figured adding cassandra into the mix as a data store for loki. Easy-peasy, I set up a two server virtualbox labtest on my macbook. With saltstack I deployed the identical installation and copied over some dashboard.
Guess what, the data is presented almost instantly (running only 1 core on the loki VM). I tweaked several performance metrics in the loki config.yaml last week so I was pretty amazed to see no performance issues on my macbook.
So will adding cassandra to the ‘live setup’ actually solve this?
Thanks a lot for any pointers and tips! 
Kind regards,
Gerard.