Promtail throwing 404

[mramuthan]

basically I am trying to ship logs from promtail agent to loki
though everthing up and running loki and promtail
for some reason my promtail agent not able to push the logs to the loki
throwing me the following error

level=error ts=2022-02-10T20:01:07.141341734Z caller=client.go:360 component=client host=ams-loki:3100 msg=“final error sending batch” status=404 error=“server returned HTTP status 404 Not Found (404): 404 page not found”

Could you please help?

[b0b]

Hi @mramuthan ,

your not giving us a whole lot to go by

Short answer, Promtail can’t find host=ams-loki:3100

How have you deployed Loki and Promtail? Based on the host name it seems like a container platform of some sort?

[mramuthan]

Here is the context, I am using the following helm charts to install loki and promtail on a k8 cluster:

1. grafana/loki (chart version: 2.9.1 , App version: v2.4.2 )
2. grafana/promtail (chart version: 3.11.0 , App version: 2.4.2 )

These charts are coming from Grafana Community Kubernetes Helm Charts | helm-charts

And following is the final configuration for the promtail agent

server:
  http_listen_network: tcp
  http_listen_address: ""
  http_listen_port: 3101
  http_listen_conn_limit: 0
  grpc_listen_network: tcp
  grpc_listen_address: ""
  grpc_listen_port: 9095
  grpc_listen_conn_limit: 0
  http_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  grpc_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  register_instrumentation: true
  graceful_shutdown_timeout: 30s
  http_server_read_timeout: 30s
  http_server_write_timeout: 30s
  http_server_idle_timeout: 2m0s
  grpc_server_max_recv_msg_size: 4194304
  grpc_server_max_send_msg_size: 4194304
  grpc_server_max_concurrent_streams: 100
  grpc_server_max_connection_idle: 2562047h47m16.854775807s
  grpc_server_max_connection_age: 2562047h47m16.854775807s
  grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
  grpc_server_keepalive_time: 2h0m0s
  grpc_server_keepalive_timeout: 20s
  grpc_server_min_time_between_pings: 5m0s
  grpc_server_ping_without_stream_allowed: false
  log_format: logfmt
  log_level: info
  log_source_ips_enabled: false
  log_source_ips_header: ""
  log_source_ips_regex: ""
  http_path_prefix: ""
  external_url: ""
  health_check_target: null
  disable: false
client:
  url: http://loki:3100/api/v1/push
  batchwait: 1s
  batchsize: 1048576
  follow_redirects: false
  backoff_config:
    min_period: 500ms
    max_period: 5m0s
    max_retries: 10
  timeout: 10s
  tenant_id: ""
  stream_lag_labels: filename
clients:
- url: http://loki:3100/api/v1/push
  batchwait: 1s
  batchsize: 1048576
  follow_redirects: false
  backoff_config:
    min_period: 500ms
    max_period: 5m0s
    max_retries: 10
  timeout: 10s
  tenant_id: ""
  stream_lag_labels: filename
positions:
  sync_period: 10s
  filename: /run/promtail/positions.yaml
  ignore_invalid_yaml: false
scrape_configs:
- job_name: kubernetes-pods
  pipeline_stages:
  - cri: {}
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_controller_name]
    separator: ;
    regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
    target_label: __tmp_controller_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name, __meta_kubernetes_pod_label_app,
      __tmp_controller_name, __meta_kubernetes_pod_name]
    separator: ;
    regex: ^;*([^;]+)(;.*)?$
    target_label: app
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component, __meta_kubernetes_pod_label_component]
    separator: ;
    regex: ^;*([^;]+)(;.*)?$
    target_label: component
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_node_name]
    separator: ;
    regex: (.*)
    target_label: node_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [namespace, app]
    separator: /
    regex: (.*)
    target_label: job
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
    separator: /
    regex: (.*)
    target_label: __path__
    replacement: /var/log/pods/*$1/*.log
    action: replace
  - source_labels: [__meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash,
      __meta_kubernetes_pod_annotation_kubernetes_io_config_hash, __meta_kubernetes_pod_container_name]
    separator: /
    regex: true/(.*)
    target_label: __path__
    replacement: /var/log/pods/*$1/*.log
    action: replace
  static_configs: []
target_config:
  sync_period: 10s
  stdin: false

I verified the loki is up and running by going to the URL http://loki:3100/ready

but the promtail agent not able to reach out to it and showing the following error

level=error ts=2022-02-10T20:01:07.141341734Z caller=client.go:360 component=client host=loki:3100 msg=“final error sending batch” status=404 error=“server returned HTTP status 404 Not Found (404): 404 page not found”

[b0b]

Looking at my own agent config I see I use this url

http://loki.loki:3100/loki/api/v1/push

I have Loki running in a separate namespace, so the service name is loki.loki. The difference I wanted to highlight is the /loki after the port which you do not have in your url. It is also in the Promtail client config comment

So I would try adding that to the url and see if that fixes the issue.