Unknown service logproto.Pusher

obiwan · June 13, 2023, 8:09pm

What does this message mean? I am seeing a lot these messages in my Loki server logs.

level=warn
  ts=2023-06-13T19:58:40.800445491Z
  caller=logging.go:86
  traceID=5e54bed20d8d9076
  orgID=fake
  msg="
    POST /loki/api/v1/push (500) 4.608829ms
    Response: \"rpc error: code = Unimplemented desc = unknown service logproto.Pusher\\n\"
      ws: false;
      Content-Length: 149668;
      Content-Type: application/x-protobuf;
      User-Agent: promtail/2.8.0;
      X-Forwarded-For: 1.2.3.4;
      X-Internal-Remote-Address: 5.6.7.8;
  "

obiwan · June 13, 2023, 8:35pm

Logs with debug mode enabled:

level=debug ts=2023-06-13T20:11:56.804599506Z caller=logging.go:76 traceID=600380131a680a68 orgID=fake msg="POST /loki/api/v1/push (204) 2.745288ms"
level=debug ts=2023-06-13T20:11:56.808077568Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/json contentEncoding= bodySize="92 kB" streams=1 entries=34 streamLabelsSize="99 B" entriesSize="80 kB" totalSize="80 kB" mostRecentLagMs=494
level=warn ts=2023-06-13T20:11:56.809891312Z caller=logging.go:86 traceID=74a875a389ed1bd9 orgID=fake msg="POST /loki/api/v1/push (500) 139.668769ms Response: \"rpc error: code = Unimplemented desc = unknown service logproto.Pusher\\n\" ws: false; Content-Length: 92203; Content-Type: application/json; User-Agent: Fluent-Bit; X-Forwarded-For: 145.145.145.145; X-Internal-Remote-Address: 145.145.145.145; "
level=debug ts=2023-06-13T20:11:56.848231901Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="375 B" streams=1 entries=1 streamLabelsSize="270 B" entriesSize="127 B" totalSize="397 B" mostRecentLagMs=1102
level=debug ts=2023-06-13T20:11:56.848947643Z caller=grpc_logging.go:46 method=/logproto.Pusher/Push duration=133.314µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:56.849290727Z caller=logging.go:76 traceID=3e0446f2e4eb6007 orgID=fake msg="POST /loki/api/v1/push (204) 1.269277ms"
level=debug ts=2023-06-13T20:11:57.192809108Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/json contentEncoding= bodySize="459 B" streams=1 entries=1 streamLabelsSize="150 B" entriesSize="236 B" totalSize="386 B" mostRecentLagMs=1499
level=debug ts=2023-06-13T20:11:57.193520693Z caller=grpc_logging.go:46 method=/logproto.Pusher/Push duration=98.708µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:57.193772373Z caller=logging.go:76 traceID=054467cfcfece672 orgID=fake msg="POST /loki/api/v1/push (204) 1.154839ms"
level=debug ts=2023-06-13T20:11:57.429355203Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/json contentEncoding= bodySize="468 B" streams=1 entries=1 streamLabelsSize="146 B" entriesSize="241 B" totalSize="387 B" mostRecentLagMs=1512
level=warn ts=2023-06-13T20:11:57.431550984Z caller=logging.go:86 traceID=10df5adb45d8cedd orgID=fake msg="POST /loki/api/v1/push (500) 2.427522ms Response: \"rpc error: code = Unimplemented desc = unknown service logproto.Pusher\\n\" ws: false; Content-Length: 468; Content-Type: application/json; User-Agent: Fluent-Bit; X-Forwarded-For: 146.146.146.146; X-Internal-Remote-Address: 146.146.146.146; "
level=debug ts=2023-06-13T20:11:57.619132428Z caller=grpc_logging.go:46 method=/grpc.health.v1.Health/Check duration=76.656µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:57.871776987Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/json contentEncoding= bodySize="115 kB" streams=1 entries=44 streamLabelsSize="99 B" entriesSize="100 kB" totalSize="100 kB" mostRecentLagMs=11281
level=warn ts=2023-06-13T20:11:57.873619346Z caller=logging.go:86 traceID=0d4dc70aac868a5b orgID=fake msg="POST /loki/api/v1/push (500) 205.267213ms Response: \"rpc error: code = Unimplemented desc = unknown service logproto.Pusher\\n\" ws: false; Content-Length: 115316; Content-Type: application/json; User-Agent: Fluent-Bit; X-Forwarded-For: 145.145.145.145; X-Internal-Remote-Address: 145.145.145.145; "
level=debug ts=2023-06-13T20:11:57.899968229Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="375 B" streams=1 entries=1 streamLabelsSize="270 B" entriesSize="127 B" totalSize="397 B" mostRecentLagMs=1100
level=debug ts=2023-06-13T20:11:57.900745389Z caller=grpc_logging.go:46 method=/logproto.Pusher/Push duration=105.791µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:57.901064777Z caller=logging.go:76 traceID=740364f45b85931a orgID=fake msg="POST /loki/api/v1/push (204) 1.275228ms"
level=debug ts=2023-06-13T20:11:58.013089295Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="380 B" streams=1 entries=1 streamLabelsSize="271 B" entriesSize="126 B" totalSize="397 B" mostRecentLagMs=1096
level=debug ts=2023-06-13T20:11:58.014127212Z caller=grpc_logging.go:46 method=/logproto.Pusher/Push duration=92.015µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:58.014349054Z caller=logging.go:76 traceID=51132f8e0db53f30 orgID=fake msg="POST /loki/api/v1/push (204) 1.434341ms"
level=debug ts=2023-06-13T20:11:58.135533585Z caller=grpc_logging.go:46 method=/grpc.health.v1.Health/Check duration=61.618µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:58.164679876Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="983 B" streams=1 entries=13 streamLabelsSize="270 B" entriesSize="2.5 kB" totalSize="2.8 kB" mostRecentLagMs=857
level=debug ts=2023-06-13T20:11:58.165555253Z caller=grpc_logging.go:46 method=/logproto.Pusher/Push duration=186.895µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:58.165921651Z caller=logging.go:76 traceID=7c5815a648745758 orgID=fake msg="POST /loki/api/v1/push (204) 1.420876ms"
level=debug ts=2023-06-13T20:11:58.440878246Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/json contentEncoding= bodySize="446 B" streams=1 entries=1 streamLabelsSize="162 B" entriesSize="207 B" totalSize="369 B" mostRecentLagMs=2824
level=debug ts=2023-06-13T20:11:58.441873271Z caller=grpc_logging.go:46 method=/logproto.Pusher/Push duration=303.358µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:58.442167531Z caller=logging.go:76 traceID=3f14efb6388a9b12 orgID=fake msg="POST /loki/api/v1/push (204) 1.477634ms"
level=debug ts=2023-06-13T20:11:58.587091638Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/json contentEncoding= bodySize="471 B" streams=1 entries=1 streamLabelsSize="148 B" entriesSize="242 B" totalSize="390 B" mostRecentLagMs=11838
level=warn ts=2023-06-13T20:11:58.588481955Z caller=logging.go:86 traceID=34d31ff439594986 orgID=fake msg="POST /loki/api/v1/push (500) 1.592343ms Response: \"rpc error: code = Unimplemented desc = unknown service logproto.Pusher\\n\" ws: false; Content-Length: 471; Content-Type: application/json; User-Agent: Fluent-Bit; X-Forwarded-For: 147.147.147.147; X-Internal-Remote-Address: 147.147.147.147; "
level=debug ts=2023-06-13T20:11:58.680486612Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="726 B" streams=1 entries=8 streamLabelsSize="278 B" entriesSize="1.5 kB" totalSize="1.8 kB" mostRecentLagMs=1037
level=debug ts=2023-06-13T20:11:58.681030107Z caller=grpc_logging.go:46 method=/logproto.Pusher/Push duration=105.891µs msg="gRPC (success)"
level=debug ts=2023-06-13T20:11:58.681228094Z caller=logging.go:76 traceID=068ba6a7f5c95d02 orgID=fake msg="POST /loki/api/v1/push (204) 877.27µs"
level=debug ts=2023-06-13T20:11:59.086681623Z caller=grpc_logging.go:46 method=/grpc.health.v1.Health/Check duration=80.964µs msg="gRPC (success)"

obiwan · June 16, 2023, 9:18pm

Seems to perhaps be related to hash rings. But I cannot confirm that yet.

obiwan · July 6, 2023, 11:08pm

I think the solution is to make sure that all of the components have an HTTP and GRPC address and port.

system · July 5, 2024, 11:09pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
POST /loki/api/v1/push (500) Response: empty ring and context canceled Grafana Loki loki	2	1211	November 18, 2023
Promtail can not push log to loki Grafana Loki loki , configuration , promtail	6	386	May 10, 2024
Learning to Post logs to Loki Grafana Loki	1	692	February 27, 2023
Promptail HTTP status 429 Grafana Loki loki	3	347	May 3, 2024
Loki logfmt cannot parse it's own gateway HTTP POST log message Grafana Loki loki , promtail	9	1311	December 16, 2023

Unknown service logproto.Pusher

Related topics