Size chunks and index difference in s3

Grafana Loki
1

Hello Loki Team,

I have a setup for Promtail, Loki, and Grafana. I have 2 VMs and 1 Dell S3 ECS bucket with a size of 300GB.

VM 1: 10.4.17.19
VM 2: 10.4.17.18

I’m currently using Loki version 2.8.2 and Standalone mode on two VM. Below is my Loki config:

auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  grpc_server_max_concurrent_streams: 1000
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  http_server_idle_timeout: 120s
  http_server_write_timeout: 1m 
common:
  path_prefix: /etc/loki/
  replication_factor: 1
  ring:
    kvstore:
      store: memberlist
  storage:
    s3:
      endpoint: https://ecs.test.net
      insecure: false
      bucketnames: bucket
      access_key_id: clickhouse_test_user
      secret_access_key: secret-key
memberlist:
  abort_if_cluster_join_fails: false
  bind_port: 7946
  join_members:
    - 10.4.17.19:7946
    - 10.4.17.18:7946
schema_config:
  configs:
    - from: "2023-01-03"
      chunks:
        prefix: chunk_
        period: 24h
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v11
      store: boltdb-shipper
    - from: "2023-06-24"
      store: tsdb
      object_store: s3
      schema: v12
      index:
        prefix: index_
        period: 24h
      chunks:
        prefix: chunk_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /etc/loki/index
    build_per_tenant_index: true
    cache_location: /etc/loki/index_cache
    shared_store: s3
  tsdb_shipper:
    active_index_directory: /etc/loki/tsdb-index
    cache_location: /etc/loki/tsdb-cache
    shared_store: s3
  aws:
    endpoint: https://ecs.test.net
    insecure: false
    bucketnames: bucket
    access_key_id: clickhouse_test_user
    secret_access_key: secret-key
    s3forcepathstyle: true
frontend_worker:
  frontend_address: 10.4.17.19:9096
  grpc_client_config:
    max_send_msg_size: 104857600
  parallelism: 9
  match_max_concurrent: true
compactor:
  working_directory: /etc/loki/
  shared_store: s3
  compaction_interval: 5m
ruler:
  storage:
    s3:
      bucketnames: bucket
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 48h
  max_global_streams_per_user: 100000
  max_entries_limit_per_query: 50000
  ingestion_rate_mb: 8190000000
  ingestion_rate_strategy: global
  ingestion_burst_size_mb: 81900000000
  max_line_size: 50000000
  per_stream_rate_limit: 100000000000
  per_stream_rate_limit_burst: 1000000000 
  query_timeout: 10m
  split_queries_by_interval: 30m
  max_cache_freshness_per_query: '10m'
query_range:
  align_queries_with_step: true
  max_retries: 5
  cache_results: true
  results_cache:
    cache:
      enable_fifocache: true
      fifocache:
        size: 10024
        validity: 24h
frontend:
  log_queries_longer_than: 3s
  max_body_size: 10485760
  query_stats_enabled: false
  max_outstanding_per_tenant: 1024
  querier_forget_delay: 0s
  scheduler_address: ""
  scheduler_dns_lookup_period: 10s
  scheduler_worker_concurrency: 5
  compress_responses: true
distributor:
  ring:
    kvstore:
      store: memberlist
ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
  chunk_idle_period: 30m
  chunk_block_size: 262144
  chunk_target_size: 1572864
  chunk_retain_period: 0s
  max_transfer_retries: 0
  chunk_encoding: gzip
  max_chunk_age: 3h
chunk_store_config:
  max_look_back_period: 0s
querier:
    engine:
        timeout: 5m
    max_concurrent: 16
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 67108864
  remote_timeout: 1s
query_scheduler:
  max_outstanding_requests_per_tenant: 32768

I have a problem that I’m trying to understand. I have a file with 5 GB size with 55,521,300 logs lines. These logs were generated using Flog and Promtail is reading the file and sending it to Loki. However, when I set the chunk encoding to either none, snappy or gzip, I end up with the same total size (2GB) in my S3 bucket. This happens regardless of the compression settings I use.

I’m trying to understand why this is the case. Any help would be greatly appreciated.

2

is this correct ?
The behavior you’re experiencing could be due to several factors.

Firstly, when Loki receives log data, it processes the data and splits it into chunks. These chunks are not direct representations of the input log data. They are processed and formatted in a specific way that makes them efficient for Loki to query. This process involves deduplication, indexing, and other operations. Therefore, the size of the chunk data in S3 may not have a direct relationship with the size of the input log data.

Secondly, the chunk encoding setting (none, snappy, or gzip) in Loki controls the encoding of the chunks that Loki creates. The chunk encoding is not meant to significantly reduce the size of the data, but rather it’s used for efficient querying and to reduce the cost of storing and transferring data. While the gzip encoding typically provides a higher compression ratio than snappy, the difference may not be significant for certain types of data, and the compression ratio can also depend on the specifics of the data being compressed.

Thirdly, the total size of the data in the S3 bucket includes not only the chunk data but also the index data. Loki uses an index to quickly locate the chunks that match a given query. The size of the index data can also contribute to the total size of the data in S3.

Therefore, the total size of the data in S3 depends on many factors and is not solely determined by the chunk encoding setting. It’s also worth noting that Loki’s primary design goal is not to minimize storage costs, but to provide fast, efficient querying of log data.

In order to understand more about this behavior, you might want to consider the following:

  1. Check the logs for Loki and Promtail for any errors or warnings. There might be issues with the processing or uploading of chunks that could give you some clues.
  2. Monitor the metrics exposed by Loki and Promtail. These can give you insights into the rate of chunk creation, the rate of chunk uploads to S3, and other aspects of the system’s behavior.
  3. Experiment with different types of log data. The size of the chunk data in S3 can depend on the specifics of the log data. For example, log data with a lot of repeated content might compress more effectively than log data with a lot of unique content.
  4. Experiment with different configurations of Loki. The behavior of Loki can be influenced by many different configuration settings. You might want to experiment with different settings to see if you can find a configuration that gives you the results you’re looking for.
  5. Consider reaching out to the Loki community or the Grafana Labs support team for help. They might be able to provide more specific advice based on their experience with similar setups.

Lastly, it’s worth noting that the metrics and logging data typically have a very different lifecycle and access pattern compared to other types of data. It’s often accessed frequently shortly after it’s generated but then rarely accessed again after a certain period of time. This makes it a good candidate for lifecycle policies in S3, where the data is moved to cheaper storage classes after a certain period of time, which can help to reduce storage costs.

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.