Hi there
I am unable to send the original log date to loki, It send only the Scrapped timestamp to loki and when I visualize in Grafana I Could see the logs assigned to current timestamp (ts=****) when the Promtail pushed the logline to loki. Enclosed the details of log, Srape config and Promtail when an o Dry run and inspect mode
Sample Log File
2021-08-11,00:00:01.771,Instruction details goes here,log details goes here.
PromTail Config File for Scrapping and timestamp stage
scrape_configs:
- job_name: test
pipeline_stages:
- regex:
expression: '(?P<time>^\d{4}-\d{2}-\d{2},\d{2}:\d{2}:\d{2}.\d{3}),(?P<State>W*(?i)Instruction(?-i)|((?i)Finished(?-i))|((?i)Started(?-i)))'
- timestamp:
source: time
format: 2006-01-02,15:04:05.999
location: "America/New_York"
- pack:
ingest_timestamp: false
Promtail in inspect mode and Dry run
Clients configured:
url: http://localhost:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
backoff_config:
min_period: 500ms
max_period: 5m0s
max_retries: 10
timeout: 10s
tenant_id: “”
stream_lag_labels: “”
level=debug ts=2022-10-18T05:48:27.025108Z caller=manager.go:265 component=discovery msg=“Starting provider” provider=static/0 subs=map[test:{}]
level=debug ts=2022-10-18T05:48:27.025108Z caller=manager.go:299 component=discovery msg=“Discoverer channel closed” provider=static/0
level=info ts=2022-10-18T05:48:27.0271078Z caller=server.go:288 http=[::]:9080 grpc=[::]:52602 msg=“server listening on addresses”
level=info ts=2022-10-18T05:48:27.0271078Z caller=main.go:121 msg=“Starting Promtail” version=“(version=2.6.1, branch=HEAD, revision=6bd05c9a4)”
level=debug ts=2022-10-18T05:48:32.0270872Z caller=filetargetmanager.go:278 msg=“new target” labels=“{address="localhost"}”
level=info ts=2022-10-18T05:48:32.0270872Z caller=filetargetmanager.go:338 msg=“Adding target” key=“…/Samplelogs/sample/*.log:{env="dev", job="test"}”
level=info ts=2022-10-18T05:48:32.0280891Z caller=filetarget.go:278 msg=“watching new directory” directory="C:\Loki\Samplelogs\sample
"
level=debug ts=2022-10-18T05:48:32.0290865Z caller=filetarget.go:318 msg=“tailing new file” filename=“C:\Loki\Samplelogs\sample\2021
0811.log”
level=info ts=2022-10-18T05:48:32.0620879Z caller=tailer.go:143 component=tailer msg=“tail routine: started” path=“C:\Loki\Samplelogs\sample\2021
0811.log”
ts=2022-10-18T05:48:32.0620879Z caller=log.go:168 level=info msg=“Seeked C:\Loki\Samplelogs\sample\20210811.log - &{Offset:0 Whence:
0}”
level=debug ts=2022-10-18T05:48:32.0630879Z caller=regex.go:132 component=file_pipeline component=stage type=regex msg=“extracted data debug in regex stage” extractedda
ta=“map[env:dev filename:C:\Loki\Samplelogs\sample\20210811.log job:test time:2022-08-11,00:00:01.771]”
[inspect: regex stage]:
{stages.Entry}.Extracted[“time”]:
+: 2022-08-11,00:00:01.771
[inspect: timestamp stage]:
{stages.Entry}.Entry.Entry.Timestamp:
-: 2022-10-18 11:18:32.0620879 +0530 IST
+: 2022-08-11 00:00:01.771 -0400 EDT
level=debug ts=2022-10-18T05:48:32.064088Z caller=output.go:75 component=file_pipeline msg=“extracted data did not contain output source”
[inspect: output stage]: none
2022-08-11T00:00:01.771-0400 {env=“dev”, filename=“C:\Loki\Samplelogs\sample\20210811.log”, job=“test”} 2021-08-11,00:00:01.771,Instruction details goes here,log details goes here.