How to push Prometheus metrics in Grafana Cloud?

Grafana Cloud
22

Support guys helped me with the question, leaving here a solution for any future wanderers.

/influx/write supports \n, but you need to properly escape it in bash, like so: ANSI-C Quoting (Bash Reference Manual)

So, the correct command to send multiple metrics in one request using curl is:
curl -d $'test,test_label=test_label_value direct_metric=22\ntest_1,test_label_1=test_label_value direct_metric=23' -i -u <username>:<key> https://influx-prod-01-eu-west-0.grafana.net/api/v1/push/influx/write

Notice $ before request body.

3 Likes
23

Hi,

In my case, the data are shown in the Prometheus datasource. Is this now the expected behaviour?

image
image1113×1019 66.3 KB

In addition, I cannot find any “influx” datasource to add to my Grafana Cloud, there is only “InfluxDB” which I assume is not the same:
image

24

Hi guys,

I have exactly the same problem like @alexsey with sending metrics to influx write

Currently we are still on free plan while evaluation.

I can not send the test values you mentioned.
foo_metric,bar_label=abc cpu_load=54.1
returns 400 bad metrics write request

If I wrap the value with double quotes
foo_metric,bar_label=abc cpu_load="54.1"
I get a 204 but can not find the metrics in Grafana with Explore.

My curl command is:

curl -v -XPOST \
-d 'test,test_label=test_label_value direct_metric=22' \
-i \
-u "<USER_FROM_PROMETHEUS_SEND_METRICS>:<API_KEY_FROM_ORG" \
https://influx-prod-01-eu-west-0.grafana.net/api/v1/push/influx/write

What am I doing wrong?

25

Hi @sbj

Can you confirm that you are trying to use Telegraf to send metrics directly to Prometheus, which is hosted in your Grafana Cloud trial?

Have you been able to send any metrics to Prometheus (hosted on Grafana Cloud)? If yes, how were they sent? Do you have plans to use any other datasources with Grafana besides Prometheus?

Just trying to figure you where you are in your journey so either me or others can help.

26

Hello @grant2,

Correct, I am trying to send metrics directly to Prometheus (part of our Grafana Cloud free plan) using curl like you described in previous posts in this thread here.

In previous posts, you already said, sending to influx in Grafana cloud should work even with free plan.

Other metrics are shown in Prometheus. But the other metrics are scraped by a grafana agent running in the Kubernetes cluster. Additionally, I want to send metrics directly because scraping does not make sense for them.

I also tried Graphite for the metrics data source. But since graphite does only accept metrics with timestamps in the past 1 hour, this does not fit perfectly our use case. We may want to push historic metrics.

27

Hi @sbj

At the risk of sounding redundant…

I use Telegraf → InfluxDB (locally hosted) → Grafana Cloud

Your setup is Telegraf ->Prometheus Cloud → Grafana Cloud

I can try to set up something simple in my test system this week and try to figure out how to work thru the error message you mentioned.

In addition to Telegraf, you also mentioned Grafana agent. Are those metrics also flowing into Prometheus, and do you have any issues with that signal chain? If it works, then is Telegraf even necessary in your setup?

Just to give you more food for thought…

  1. Influx DB also has a cloud product that can be used with Grafana.
  2. Is there any reason you have chosen Prometheus as your database instead of InfluxDB? Both are excellent.
  3. Is all of this data coming from K8s, or is there other data elsewhere that you plan to scrape?
28

Hello @grant2

No, incorrect. I said nothing about Telefgraf, nor InfluxDB.

I want to send metrics directly from my application (any client sending http request) to prometheus to store custom metrics, that cannot be scraped (because the app is not available via http).

My setup is client with http request / curl -> Prometheus Cloud -> Grafana Cloud

The whole thread here is about sending metrics directly to Prometheus, without scraping.

The documentation at Push metrics from Influx Telegraf to Prometheus | Grafana Cloud documentation just does not work for me. If you look at the examples, the node.js example says you need to set 'Content-Type': 'application/json', but the python example says 'Content-Type': 'text/plain',

Following the documentation, sending an HTTP request with body:
foo_metric,bar_label=abc cpu_load=54.1
returns 400 bad metrics write request

If I wrap the value with double quotes
foo_metric,bar_label=abc cpu_load="54.1"
I get a 204 but cannot find the metrics in Grafana with Explore.

29

Got it. Thanks for explaining. I will see what I can find out when I am at my test system.

1 Like
30

Is your app a nodejs app?

31

No, only javascript but not with node.js environment. The app gets triggered 4-8 times a day and shall collect metrics in the past 4 hours and push them to Prometheus. Mainly an offline use case.

2 Likes
32

Hey everyone,

I might found the root issue of my problems with the example of pushing metrics into Prometheus.

I just saw that we already reached the limit of 10k metrics in our free plan. Somehow, we are sending lots of different metrics with our Kubernetes cluster installation.

With the knowledge of reaching the limit, the error 400 bad metrics write request makes sense.
I just tested it with a colleague in a different account and the request is accepted.

So my task for tomorrow will, either start the paid subscription without having the limit of 10k metrics or reduce the send metrics.

But still strange that,
foo_metric,bar_label=abc cpu_load="54.1"
returns a 204, but you cannot find the metrics in Grafana with Explore, then.

2 Likes
33

Thanks, I was reading this thread, and here your example is the one that worked for me and now I’m able to see the value in grafana.

2 Likes
34

I fully failed to push from rust to official default cloud (so I made fixes for snappy and url builing)

35

metrics/prometheus_push_gateway.rs at main · metrics-rs/metrics · GitHub < this one has not basic auth and https support. found tab how to curl push. above solution sure sends wrong format.