What is the correct way to parse json logs in loki, promtail

[sdeoras]

Hi Folks, I am trying to use loki and not able to properly configure promtail to parse JSON logs. Everything is on a k8s cluster. The log structure is a JSON string without any nesting.

{"level":"info","time":"2021-08-16T18:26:46.621Z","name":"bme280.data","msg":"data","temperature":26.03,"pressure":999.39,"humidity":45.32,"altitude":116}

I used following promtail config:

    scrape_configs:
    - job_name: kubernetes-pods-name
      pipeline_stages:
        - json:
            expressions:
              altitude: altitude
              humidity: humidity
              level: level
              output: msg
              pressure: pressure
              temperature: temperature
              timestamp: time
        - labels:
            level: null
        - timestamp:
            format: RFC3339
            source: timestamp
        - output:
            source: output
      kubernetes_sd_configs:
      - role: pod

I was hoping that simply by defining the JSON parsing and labels I get to see these labels in the Grafana Explore. I do see the log entries in Grafana as shown below, but I don’t see any labels that are part of the JSON keys.

Any pointers on how to make this work? Apprciate any help, thanks!

[dannykopping]

Hey @sdeoras

If you look closer, you’ll see your log message is prefixed with stdout F.
You should run your logs through the cri pipeline stage (docs) before attempting to parse the JSON.

[sdeoras]

hey @dannykopping , that worked! thanks.

[dannykopping]

@sdeoras happy it worked for you!
Please mark my answer as the solution so others can find it easily.

[virtualb0x]

@sdeoras can you be so kind to paste the config where after cri the JSON is parsed?

[gabrielcerutti1]

@sdeoras Could you please post the final config? I’m trying to achieve the same but still not parsing the json successfully

[jerylcook]

You just have to add it as another ‘pipeline stage’ like this. its regex pattern.
you should already have a “kubernetes-pods-name”

scrape_configs:
- job_name: kubernetes-pods-name
  pipeline_stages:
    - json:
        expressions:
          altitude: altitude
          humidity: humidity
          level: level
          output: msg
          pressure: pressure
          temperature: temperature
          timestamp: time
	- regex:
		expression: "^(?s)(?P<time>\\S+?) (?P<stream>stdout|stderr) (?P<flags>\\S+?) (?P<content>.*)$"
	- labels:
		stream:
	- timestamp:
		source: time
		format: RFC3339Nano
	- output:
		source: content
    - labels:
        level: null
    - timestamp:
        format: RFC3339
        source: timestamp
    - output:
        source: output
  kubernetes_sd_configs:
  - role: pod