Promtail fails to send logs to Grafana cloud: has 18 label names; limit 15

Noticed that ambassador edge-stack pods logs are not being sent over and promtail shows up logs:

promtail-hnp92] level=error ts=2022-05-02T14:27:37.17420237Z
caller=client.go:294 component=client host=*.grafana.net:443 msg="final error
sending batch" status=400 error="server returned HTTP status 400 Bad Request
(400): {\"code\":400,\"status\":\"error\",\"message\":\"entry for stream
'{app_kubernetes_io_instance=\\\"edge-stack\\\",
app_kubernetes_io_managed_by=\\\"Helm\\\",
app_kubernetes_io_name=\\\"edge-stack\\\",
app_kubernetes_io_part_of=\\\"edge-stack\\\",
asdf_id=\\\"xxxxxxxxxxxxxxxxxxxxxxxx\\\", container=\\\"emissary-ingress\\\",
controller_revision_hash=\\\"79c7ddd8fc\\\", environment_name=\\\"staging\\\",
filename=\\\"/var/log/pods/ambassador_edge-stack-7b5sg_50346ab3-2ec7-4623-8ddc-a4991b53b8c0/emissary-ingress/0.log\\\",
helm_sh_chart=\\\"emissary-ingress-7.2.0\\\",
job=\\\"ambassador/edge-stack\\\", k8s_cluster=\\\"xxxxxxxxxxxxx\\\",
namespace=\\\"ambassador\\\", pod=\\\"edge-stack-7b5sg\\\",
pod_template_generation=\\\"3\\\", product=\\\"aes\\\", profile=\\\"main\\\",
stream=\\\"stderr\\\"}' has 18 label names; limit 15\"}"

Also has pods with verious other labels/dimentions assigned like istio-related sidecar containers.

Any way to filter out that dimensions?

Hello! You can use relabel_configs to selectively discard or drop certain labels, though I’m not too sure where the istio-related labels are coming from. Was the Promtail configuration created using the Integrations and Connections tool in your Grafana Cloud account?

1 Like

@Melody thanks. Already went ahead and added droplabel stage in promtail config. Which works for now as expected for us.

1 Like

@Melody, thanks for your reply. I’ve faced a similar issue, and although I can drop some labels, I’m afraid I might need to increase the label names in the future. So I wonder if it’s possible to increase the limit?