Err="rpc error: code = Code(400) desc = entry with timestamp 2022-07-18 07:26:08.685550897 +0000 UTC ignored, reason: 'entry too far behind' for stream

[suresh300567]

Hi
We are ingesting logs from Fluentd to loki and loki is running on K8 enviroment with replication value 3 .

We are getting the below error - err="rpc error: code = 2022-07-21T21:04:36.428474528+02:00 level=warn ts=2022-07-21T19:04:36.428343978Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=145.357µs err=“rpc error: code = Code(400) desc = entry with timestamp 2022-07-18 07:26:08.685550897 +0000 UTC ignored, reason: ‘entry too far behind’ for stream: {container="XXXXXX", env="XXXXXXX", fluentd_thread="flush_thread_1", namespace="kube-system", pod="XXXXXXXX"},\nentry with timestamp 2022-07-18 07:26:59.909540823 +0000 UTC ignored, reason: ‘entry too far behind’ for stream: {container="cXXXXXX", env="XXXXXX", fluentd_thread="flush_thread_1", namespace="kube-system", pod="XXXXXXX"},\ntotal ignored: 2 out of 2” .

Please any suggestions would be very helpful .
I have seen here that loki accepts out of order entries by default and depends on max_chunk_age value. Any inputs would be really helpfull.
----- As per loki config documentation --------------
How far into the past accepted out-of-order log entries may be is configurable with max_chunk_age. max_chunk_age defaults to 1 hour. Loki calculates the earliest time that out-of-order entries may have and be accepted with

time_of_most_recent_line - (max_chunk_age/2)

Log entries with timestamps that are after this earliest time are accepted. Log entries further back in time return an out-of-order error.

Our Loki config file -
loki:

– Config file contents for Loki

@default – See values.yaml

config: |
auth_enabled: [% loki_config.auth_enabled %]

target: all,table-manager

server:
  http_listen_port: [% loki_config.http_listen_port %]
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 16777216

distributor:
  ring:
    kvstore:
      store: memberlist

memberlist:
  join_members:
    - {{ include "loki.fullname" . }}-memberlist


ingester:
  # Disable chunk transfer which is not possible with statefulsets
  # and unnecessary for boltdb-shipper
  max_transfer_retries: 0
  chunk_idle_period: 1h
  chunk_target_size: 1536000
  max_chunk_age: 8h
  lifecycler:
    join_after: 1s
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
  wal:
    enabled: false

ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 16777216

query_scheduler:
  max_outstanding_requests_per_tenant: 2048
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 16777216

query_range:
  parallelise_shardable_queries: false


limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_cache_freshness_per_query: 10m
  max_query_lookback: 2352h		

schema_config:
  configs:

{% if loki_cassandra.enabled %}
- from: [% loki_cassandra.from_before %]
store: cassandra
object_store: cassandra
schema: v11
index:
prefix: [% loki_cassandra.index.prefix %]
period: [% loki_cassandra.index.period %]
chunks:
prefix: [% loki_cassandra.chunks.prefix %]
period: [% loki_cassandra.chunks.period %]
{% endif %}
- from: 2020-09-07
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: loki_index_
period: 24h

storage_config:
  boltdb_shipper:
    shared_store: s3
    active_index_directory: /var/loki/index
    cache_location: /var/loki/cache
    cache_ttl: 168h
  aws:
    # Note: use a fully qualified domain name, like localhost.
    # full example: http://loki:supersecret@localhost.:9000
    s3: "[% loki_minio.connection_string %]"
    s3forcepathstyle: true
    http_config:
      insecure_skip_verify: true
  index_queries_cache_config:
    memcached:
      batch_size: 100
      parallelism: 100
    memcached_client:
      consistent_hash: true
      host: {{ include "loki.memcachedIndexQueriesFullname" . }}
      service: http

{% if loki_cassandra.enabled %}
cassandra:
auth: true
username: [% loki_cassandra.username %]
password: [% loki_cassandra.password %]
addresses: [% loki_cassandra.addresses %]
keyspace: [% loki_cassandra.keyspace %]
consistency: [% loki_cassandra.consistency %]
{% endif %}
chunk_store_config:
max_look_back_period: 2352h

table_manager:
  chunk_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  index_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  retention_deletes_enabled: true
  retention_period: 2352h

query_range:
  align_queries_with_step: true
  max_retries: 5
  split_queries_by_interval: 15m
  cache_results: true
  results_cache:
    cache:
      memcached_client:
        consistent_hash: true
        host: {{ include "loki.memcachedFrontendFullname" . }}
        max_idle_conns: 16
        service: http
        timeout: 500ms
        update_interval: 1m

query_scheduler:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 16777216
  


frontend_worker:
  frontend_address: {{ include "loki.queryFrontendFullname" . }}:9095
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 16777216

frontend:
  log_queries_longer_than: 5s
  compress_responses: true
  tail_proxy_url: http://{{ include "loki.querierFullname" . }}:3100

compactor:
  shared_store: s3