How does the `timestamp` configuration in Loki work?

[bharold]

I am attempting to configure Pomtail to send historical logs to Loki. The logs are in JSON format, but I can also get them in CSV or XML. I can change pretty much anything I need to about the logs. This is a code coverage report, so it’s just key-value pairs including a timestamp of when the report was run.

I’ve tried supplying the timestamp in both UNIX and RFC3339 formats. For the UNIX timestamp I supplied it as both a string and an integer. I’ve attempted to use the following formats:

• unix
• “unix”
• Unix
• “Unix”
• 1562708916
• “1562708916”

I thought maybe there was some problem parsing Unix timestamps so I switched to RFC3339 format but I’m still not getting anywhere.

My current job config looks like this:

- job_name: loc
  pipeline_stages:
  - json:
      expressions:
        timestamp: "time"
        loc: "loc"
        cloc: "cloc"
  - labels:
      timestamp:
      loc:
      cloc:
  - timestamp:
      source: timestamp
      format: "DATE_RFC3339"
  static_configs:
  - targets:
      - localhost
    labels:
      job: hosted
      __path__: /logs/loc.log

The (simplified) JSON schema looks something like this:

{
  "loc": 1099687,
  "cloc": 119713,
  "time": "2019-02-14T00:00:00+00:00"
}

At first I thought that I was having problems because the timestamp was being loaded under “detected fields” instead of “log labels” in the Grafana interface, but after messing with it for several hours, I finally got timestamp to show up under “log labels” by adding the - labels pipeline stage configuration, but the timestamp configuration still does not seem to do anything.

Googling this problem yields a lot of different weird edge cases including using double quoted strings for some fields. Is there any example of a working implementation of using a timestamp from a JSON payload instead of using the timestamp that Grafana automatically attaches to the log?

[alexisbel]

Is there any update on this topic? I face the same issue.

[mattixpet]

I have the exact same issue, I have some logs with their own timestamp. I’ve managed to extract them as timestamp labels, but grafana still always shows the time as the time I got the logs, so that huge file as one timestamp. I still haven’t figured out how to use those timestamp labels to get a timeseries of the logs. Did you manage to figure this out? Or can anyone help?

Why is it so hard to swap out Loki’s automatic timestamps for log timestamps?

[alextes]

For me the timestamp extracts just fine, but now all the other fields in the log are gone .

        pipeline_stages:
          - cri: {}
          - match:
              selector: '{app="supply-deltas-sync", container="geth-supply-deltas"}'
              stages:
              - json: 
                  expressions:
                    level: lvl
                    output: msg
                    timestamp: t
              - labels:
                  level:
                  message: output
                  other_timestamp: timestamp
              - timestamp:
                  source: timestamp
                  format: RFC3339Nano
              - output:
                  source: output

[john4bbf]

I finally got it working! Thanks to everyone in this thread, you got me to the one yard line.

In our case, the date format was a little off. I had to figure out the Go date formats - and it’s quite literal, the mythical ‘2006-01-02 15:04:05 -0700’

    pipeline_stages:
      - multiline:
            firstline: '^\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s[+-]\d{4}\]\s'
            max_lines: 512
            max_wait_time: 3s
      - regex:
            expression: '\[(?P<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s[+-]\d{4})\]\s\[(?P<procid>\d*)\]\s\[(?P<severity>.*)\]'
      - labels:
            # these have to be parsed from the regex, otehrwise list
            # static labels in 'labels' above
            time:
            severity:
            procid:

      - timestamp:
            # [2020-08-21 18:37:05 +0000] [3366] [INFO] Booting worker with pid: 3366
            source: time
            # this is a weird Go format. See:
            # https://grafana.com/docs/loki/latest/clients/promtail/stages/timestamp/
            format: '2006-01-02 15:04:05 -0700'

This allowed us to import some historical logs. I found several articles that said historical logs can’t be parsed, but for me the solution was:

1. Startup Promtail
2. Shutdown Promtail
3. Edit the ‘positions’ file
4. Change any offsets for logs you want historical information on to “0” and save.
5. Restart