Grafana queries aren't working as expected

Grafana
,
1

Prometheus works fine, the issue is when I run a query with grafana

This works:

max(node_filesystem_free{instance=“10.0.2.242:9100”,job=“node_exporter_staging”})

But I’m trying to place each server in it’s own “job” so when I run the query in Grafana I can use job=“blah”}) and it pull stats just from that server without the IP

For example:

I have job name: staging-api-s

I can’t run:
max(node_filesystem_free{job=“staging-api”})

Even if I use group:
max(node_filesystem_free{group=“staging-api”})

I am trying to get it to pull stats based on job=“job_name”

Here’s my prometheus.yml:

global:
scrape_interval: 10s
evaluation_interval: 5s

external_labels:

monitor: ‘staging-monitor’

scrape_configs:

  • job_name: prometheus
    static_configs:
    • targets: [‘10.0.129.106:9090’]
      labels:
      instance: prometheus
  • job_name: rabbitmq
    static_configs:
    • targets: [‘10.0.1.60:9100’]
      labels:
      instance: staging-rabbitmq
  • job_name: staging-cron
    static_configs:
    • targets: [‘10.0.1.121:9100’]
      labels:
      instance: staging-cron
  • job_name: staging-api-s
    static_configs:
    • targets: [‘10.0.1.244:9100’]
      labels:
      instance: staging-api-s
  • job_name: staging-instance-s
    static_configs:
    • targets: [‘10.0.2.242:9100’]
      labels:
      instance: staging-instance-s
  • job_name: staging-api-s2
    static_configs:
    • targets: [‘10.0.2.160:9100’]
      labels:
      instance: staging-api-s2
  • job_name: staging-instance-s2
    static_configs:
    • targets: [‘10.0.1.4:9100’]
      labels:
      instance: staging-instance-s2
  • job_name: staging-haproxy1-s
    static_configs:
    • targets: [‘10.0.1.208:9100’]
      labels:
      instance: staging-haproxy1-s
  • job_name: staging-haproxy2-s
    static_configs:
    • targets: [‘10.0.2.49:9100’]
      labels:
      instance: staging-haproxy2-s
2

This doesn’t seem to be related to Grafana. But rather how you choose to structure your data.

If you go your Prometheus server at http://prometheus:9090/graph and write your queries. Do you get what your expecting?