Grafana Cloud Private Probe Configuraton Issue - GRPC

Grafana Cloud
1

Hi,

I’m trying to set up Private Probe by deploying Docker image to Google CloudRun following this documentation - Private probes | Grafana Labs

I was able to successfully deploy the image but keep encountering a GRPC error.

Steps taken:

  • Ensured that Synthetic monitoring is installed
  • Ensured container port is 4050
  • Ensured Private Probe Authentication token is correct
  • Ensured that Probe API Server URL region is correct.

Our region is eu-west and I’ve tried multiple times with different URLs as the one in the documentation did not work.

URLs I have tried:
synthetic-monitoring-grpc-eu-west.<COMPANY>.grafana.net:443
synthetic-monitoring-grpc-eu-west.grafana.net:443
synthetic-monitoring-grpc-eu-west.grafana.net

when using the URL from the documentation - synthetic-monitoring-grpc-eu-west.<COMPANY>.grafana.net:443 - I get the below error

Starting HTTP server on :4050...
shutting down
E: dialing GRPC server synthetic-monitoring-grpc-eu-west..<COMPANY>.grafana.net:443: context canceled

Would anyone able to point out what would be wrong with the settings?

Thanks and any help is appreciated!

2

Hi @youngjunpark,

For eu-west region, synthetic-monitoring-grpc-eu-west.grafana.net:443 is correct URL.

Can you try running this probe locally with debug logs enabled, and see if can connect and register with Sunthetic Monitoring API?

docker run grafana/synthetic-monitoring-agent --api-server-address=synthetic-monitoring-grpc-eu-west.grafana.net:443 --api-token=${API_TOKEN} --verbose=true --debug should do the trick.

API_TOKEN is Synthetic Monitoring Private Probe Access token, the token you got when you created the private probe.

also see Troubleshooting Private Probes

If it fails to connect, can you share debug logs with us?

1 Like
3

Hi @surajsidh,

Thank you for the suggestion. I have tried running the probe locally using the exact same command as you suggested, and I can see the probe being registered and becoming online

However, when deploying this on GCP Google Cloud Run, I get an error with below logs:

fetching check configuration from synthetic-monitoring-api
invalid authorization encoding
handling check changes
fetching check configuration from synthetic-monitoring-ap
...

The configuration is the same but doesn’t seem to work on GCP.

Would you know what could be causing the error invalid authorization encoding?

4

As you mentioned, it works on local machine that shows that you have a valid token.

"invalid authorization encoding" means that API_TOKEN is not correctly encoded, and API is not able to decode and process it.

check if the synthetic-monitoring-agent process in GCP Google Cloud Run is getting the same token that you see locally.

maybe it’s being token is modified somewhere before it’s passed on to synthetic-monitoring-agent process in GCP Google Cloud Run.

2 Likes
5

Thanks for the tip! Really appreciate it.

The probe is online after fixing the token issue as you suggested. However, I noticed a few strange things. First, the same log messages are still showing up even after probe is online

Also, both on GCP and local, the reachability is N/A. Would this be an expected behaviour?

1 Like
6

all the probes that are not running any checks will show it as N/A

Reachability on the probes page shows the reacability of checks running on that probe, you can assign some checks to the probe, and reachability will be updated after few minutes.

fetching check configuration from synthetic-monitoring-api and
handling check changes are normal log lines, and expected.

other errors might be intermittent, should be fine as long it’s online and publishing data from the checks running it.

You can monitor /metrics endpoint for metrics(port 4050 by default) of the agent to know more about health of the probe.

2 Likes
7

Thanks so much @surajsidh ! It is working great! Really appreciate all the help

1 Like