Hi!
I use LogQL to fetch logs from an external Loki server (via /loki/api/v1/query_range endpoint ). The rate of logs is about ~10 entries per second, not that high. Logs are not formatted, they are just raw messages that Python’s logging module produces, and I use queries to fetch some data and construct a report table, so it’s important that every bit of information is available.
Among all possible logs, there are log strings that look like the following one:
[2021-05-22 06:28:07].00716 : module : INFO : Processing request id <a long ID>
The problem is that sometimes the same message can be fetched when searching for the “Processing request” message (via |= operator), but it cannot be found if we use the exact ID (via the same operator or regexp parser). I found that if small time frames (like 5m difference between start and end timestamps) are used, then the probability that a given line would be returned by Loki is higher.
Does Loki really skip logs in favor of optimizations? If so, is there any way to prevent it? How can we guarantee that all required information is returned?
Thanks for any hints!