OpenTelemetry endpoint of Grafana Cloud

Grafana Cloud
1

Hello,

I was trying to configure my application to send data to Grafana Cloud using OpenTelemetry protocol according to instructions at Send data using OpenTelemetry Protocol (OTLP) | Grafana Cloud documentation. So I configured my application to publish to specified endpoint (https://otlp-gateway-prod-us-central-0.grafana.net/otlp) using specified protocol (http/protobuf) and specified authentication (Authorization=Basic base64(instace:key)). But for endpoint generated according to that page my application consistently gets 404 error.
Are there any hidden/missing settings which need to be turned on?

2

Hi @aleksandrkonstantino! I will attempt to help you troubleshoot the issue. Can you provide more details of your config and any logs you are seeing?

3

Hello,

Thanks a lot for response. I’m using OpenTelemetry Protocol Exporter with .NET code. The configuration adds exporter like this:
AddOtlpExporter(o =>
{
o.Endpoint = new Uri(cfg.Endpoint);
o.Protocol = OtlpExportProtocol.HttpProtobuf;
o.Headers = cfg.Headers;
});
}
Here cfg.Endpoint = “https://otlp-gateway-prod-us-central-0.grafana.net/otlp” and cfg.Headers = “Authorization=Basic NjE2…MwPQ==”.

When trying to report metrics and traces the code throws exception

System.Net.Http.HttpRequestException
HResult=0x80131500
Message=Response status code does not indicate success: 404 (Not Found).
Source=System.Net.Http
StackTrace:
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()

This exception was originally thrown at this call stack:
System.Net.Http.dll!System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() Unknown No symbols loaded.
OpenTelemetry.Exporter.OpenTelemetryProtocol.dll!OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient<OpenTelemetry.Proto.Collector.Trace.V1.ExportTraceServiceRequest>.SendExportRequest(OpenTelemetry.Proto.Collector.Trace.V1.ExportTraceServiceRequest request, System.Threading.CancellationToken cancellationToken) Unknown No symbols loaded.
OpenTelemetry.Exporter.OpenTelemetryProtocol.dll!OpenTelemetry.Exporter.OtlpTraceExporter.Export(OpenTelemetry.Batch<System.Diagnostics.Activity> activityBatch) Unknown No symbols loaded.
OpenTelemetry.dll!OpenTelemetry.BatchExportProcessor<System.Diagnostics.Activity>.ExporterProc() Unknown No symbols loaded.
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown No symbols loaded.

4

Hello,
is there an update on the issue? I’m facing the same problem on the “prod-eu-west-0” instance.

5

From the stacktrace, this appears to be tracing instrumentation. Can you try appending /v1/traces to the URL?
https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/traces

2 Likes
6

@robertlankford Thank you so much. This solved the issue for me.
After appending /v1/traces to the URL I got a “401 Unauthenticated” error because the library I use (opentelementry-otlp for Rust) does not yet support OTEL_EXPORTER_OTLP_HEADERS. Configuring everything in code solved that issue too, and now it works :slight_smile:

7

Works for me too. No more exceptions. And for metrics I added /v1/metrics.

8

I am also using Rust opentelemetry-otlp, but I am stuck with 401 Unauthenticated. Here is my config:

    let ot_auth = "Basic <base64(instance_id:key)>";
    let headers = HashMap::from([("Authorization".into(), ot_auth.into())]);
    let exporter = opentelemetry_otlp::new_exporter()
        .http()
        .with_endpoint("https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces")
        .with_headers(headers);

Any idea what might be wrong @biwecka ?