Hello,
I have a couple of general questions for which I could not find an answers.
I have a kubernetes cluster and I want to collect the logs from the apps running on it. I have installed promtail and loki (as a single binary, not distributred) on the cluster and promtail is sending logs to loki and loki sends them to S3 on AWS.
And I have configured loki and grafana on my home pc to look the logs from the S3.
When the cluster looses connection to the S3 I saw that the logs are buffered because when I recovered the connection after 15-20min the logs from this interval were sent to the S3. But during this time the memory used by loki increased from 50mb to 100mb.
- Are the chunks or indexes stored in the memory during time when the internet connection is missing?
- Can these buffered chunks and indexes be stored on the hard disk? Because in my case there can be days without internet connection and I don’t want to loose logs nor loki to use a lot of memory ( I want to keep set a limit of loki around 150mb for example and not to fail with out of memory, every buffering to happens on the local storage). Now I removed the connection from about an hour and a half and loki now is using 500mb and that number continues to go up…
- Is my setup ok from architecture point of view? I saw that it’s recommended to use distributed loki on central machine and send logs there directly there from promtail but makes the things a lot more complicated for me.
I am expecting max 1GB logs per day.
This is my Loki configuration:
config:
auth_enabled: false
ingester:
chunk_idle_period: 3m
chunk_block_size: 262144
chunk_retain_period: 1m
max_transfer_retries: 0
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
- from: 2021-02-20
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
server:
http_listen_port: 3100
storage_config:
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: s3
aws:
# aws credentials
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
compactor:
working_directory: /data/loki/boltdb-shipper-compactor
shared_store: s3