Loki mixin: LokiRequestLatency alert got fired

Hi everyone, recently we just enabled the Loki mixin from upstream: loki/production/loki-mixin at main · grafana/loki · GitHub, and LokiRequestLatency (loki/alerts.libsonnet at main · grafana/loki · GitHub) gets fired constantly since the p99 latency of querier and query frontend for my environment sometimes is around 4s.
My questions are mainly:

  1. How to solve this permanently? does scale up/out querier/query-frontend help?
  2. The severity of this alert is critical, does it deserve it? or warning should be OK?

Thanks!

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.