How to best handle + 6500 Log files in one job?

Grafana Loki
1

Hi,

I’m evaluating Loki (single instance in Docker) for ingesting logs from 6 mail servers (MS Exchange).
Currently I have more than 6500 log files stored locally and get following error from Promtail:

level=warn ts=2022-02-21T07:49:32.280512281Z caller=client.go:349 component=client host=loki:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased"

I have tried increasing limits in my Loki config. (See full configs below)

I’m not sure how to reduce labels as I only configured one job? I set the path to the localfiles using file globbing. Is this a mistake?

__path__: /var/log/loki/exchange/*/*/*log

Does Loki create a label per file like this? (Resulting in 6500+ labels?)
What’s the best way for my usecase?

Any help is much appreaciated.

This is my Promtail config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: exchange
  static_configs:
  - targets:
      - localhost
    labels:
      job: exchange
      __path__: /var/log/loki/exchange/*/*/*log

Loki config:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# https://grafana.com/docs/loki/latest/configuration/#limits_config
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 12
  ingestion_burst_size_mb: 24
  per_stream_rate_limit: 24MB
chunk_store_config:
  max_look_back_period: 336h
table_manager:
  retention_deletes_enabled: true
  retention_period: 2190h
ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_encoding: snappy
2

This is a culprit here… Should be: __path__: /var/log/loki/exchange/*/*/*.log You forgot dot ( . ) sign between last asterisk ( * ) and file extension ( log )

1 Like
3

Thanks, didn’t see that.

1 Like
4

Looks like that solved my main issue with ingestion. Thanks again. I now face another problem with queries. Querying over logs (6500 files) for e.g. for an IP address crashes Loki.
Example Query: {job="exchange"} |= ip("XXX.XXX.X.XXX")

I’m running Loki in Docker on a host with 6 cores and 32GB Ram. As soon as I query CPU spikes to maximum and Loki crashes. Any idea if a query over that many files should be feasable using Loki in single binary mode in Docker? Anything more I can optimize in my configs?

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.