Hey there,
We are moving our loki environment to docker to kubernetes and I am having trouble getting any data to show up.
Our logging pipleine goes from fluent-bit to fluentd to loki installed with the helm chart running as a single service. There are no errors in any of the logs as far as I can see. However nothing is being written to our s3 bucket and in the loki logs I am seeing ‘msg=“Get - deadline exceeded” key=collectors/ring’. slightly larger stacktrace for reference:
level=debug ts=2021-09-30T17:52:11.34601498Z caller=mock.go:159 msg=Get key=collectors/ring modify_index=280 value=“"\xa5\x05\xf4\xa4\x02\n\xa2\x05\n\x06loki-0\x12\x97\x05\n\x0f10.0.31.32:9095\x10\xc6\xf2\u05ca"”
level=debug ts=2021-09-30T17:52:11.34606237Z caller=mock.go:86 msg=CAS key=collectors/ring modify_index=280 value=“"\xa5\x05\xf4\xa4\x02\n\xa2\x05\n\x06loki-0\x12\x97\x05\n\x0f10.0.31.32:9095\x10\xcb\xf2\u05ca"”
level=debug ts=2021-09-30T17:52:11.346094395Z caller=mock.go:159 msg=Get key=collectors/ring modify_index=281 value=“"\xa5\x05\xf4\xa4\x02\n\xa2\x05\n\x06loki-0\x12\x97\x05\n\x0f10.0.31.32:9095\x10\xcb\xf2\u05ca"”
level=debug ts=2021-09-30T17:52:11.34612015Z caller=mock.go:113 msg=Get key=collectors/ring wait_index=281
level=debug ts=2021-09-30T17:52:13.342450631Z caller=mock.go:149 msg=“Get - deadline exceeded” key=collectors/ring
level=debug ts=2021-09-30T17:52:13.342503547Z caller=mock.go:113 msg=Get key=collectors/ring wait_index=281
level=debug ts=2021-09-30T17:52:14.342626282Z caller=mock.go:149 msg=“Get - deadline exceeded” key=collectors/ring
level=debug ts=2021-09-30T17:52:14.342676166Z caller=mock.go:113 msg=Get key=collectors/ring wait_index=281
level=debug ts=2021-09-30T17:52:15.342852695Z caller=mock.go:149 msg=“Get - deadline exceeded” key=collectors/ring
We have an iam policy with s3:* assigned to the service account and a bucket policy set up on our s3 bucket.
we are running the loki helm chart version 2.6.0.
Here are the values for our helm chart:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<Our account id>:role/<s3 role id>
extraArgs:
log.level: debug
config:
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h
chunk_retain_period: 30s
max_transfer_retries: 0
max_chunk_age: 1h
chunk_target_size: 1048576
schema_config:
configs:
- from: 2018-04-15
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: loki_index
period: 24h
storage_config:
aws:
s3: s3://ca-central-1/<bucket-name>
boltdb_shipper:
active_index_directory: /data/loki/index
shared_store: s3
cache_location: /data/loki/boltdb-chache
cache_ttl: 24h
compactor:
working_directory: /data/loki/boltdb-shipper-compactor
shared_store: aws
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: true
retention_period: 720h
Any insights would be GREATLY appreciated. I have been spinning my wheels on this for a while.