Graph out unique users based on nginx logs

Grafana Elasticsearch
1

I’ve an Elasticsearch installation which collects nginx’s logs which I need to visualize in Grafana in the following way:

top browsers which access certain url. let's suppose index.html
count opens of certain web page counting same ip and user agent as one

Data Structure stored in elasticsearch is same to

"_index" : "redacted",
   "_type" : "_doc",
   "_id" : "redacted",
   "_score" : 0.9736392,
   "_source" : {
     "upstream_status" : "302",
     "@timestamp" : "2019-11-26T03:02:08.754Z",
     "http_referer" : "-",
     "ecs" : { },
     "body_bytes_sent" : "0",
     "host" : { },
     "time_local" : "26/Nov/2019:03:02:08 +0000",
     "the_real_ip" : "redacted",
     "message" : "redacted - [redacted] - - [26/Nov/2019:03:02:08 +0000] \"GET / HTTP/2.0\" 302 0 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36\" 220 0.005 [production-webadmin-80] redacted:8080 0 0.004 302 979897183c421507eaea8346eeb5904c",
     "proxy_upstream_name" : "production-webadmin-80",
     "remote_user" : "-",
     "service" : "ingress-nginx.ingress-nginx-internal",
     "request_time" : "0.005",
     "upstream_response_time" : "0.004",
     "tags" : [
       "_geoip_lookup_failure"
     ],
     "upstream_addr" : "redacted:8080",
     "user_agent" : {
       "device" : "Other",
       "patch" : "3865",
       "os_major" : "10",
       "build" : "",
       "minor" : "0",
       "name" : "Chrome",
       "major" : "77",
       "os" : "Mac OS X",
       "os_name" : "Mac OS X",
       "os_minor" : "14"
     },
     "geoip" : { },
     "kubernetes" : {
       "pod" : {
         "name" : "redacted-internal-76dd64fb5f-stj99"
       },
       "container" : {
         "name" : "redacted"
       },
       "labels" : {
         "app" : "ingress-nginx-internal",
         "appId" : "ingress-nginx"
       },
       "namespace" : "ingress-nginx",
       "replicaset" : {
         "name" : "redacted-internal-76dd64fb5f"
       },
       "node" : {
         "name" : "redacted-20190904071-128f9703-gnvr"
       }
     },
     "request" : "GET / HTTP/2.0",
     "http_user_agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36",
     "stream" : "stdout",
     "request_length" : "220",
     "upstream_response_length" : "0",
     "status" : "302",
     "input" : { }

Now I’ve no idea how to do this using grafana. one thing I was able to get working was a query like

request : index.html AND status : 200

Could anybody help?

2

Use group by like below:

image
image778×82 7.53 KB

Group by Terms: user_agent.name Top 10
Group by Terms: the_real_ip Top 10

You can add Group by Terms as more as you can

Fadjar Tandabawana