How can I tell Loki to remove a reference to a file it no longer has access to?

[spencerdcarlson1]

This is a cross post of my question on the closed issue 429.

I had the grafana-agent reading from a file that it did have access to but now does not. I updated my job to read the log from a new location which is working as expected, but the agent is attempting to update metadata about the prior log and filling up my sys logs. How can I get loki to completely forget about the previous file?

Here is the log output:

Jul  6 15:03:33 localhost grafana-agent[11276]: ts=2023-07-06T15:03:33.342184934Z caller=positions.go:206 level=warn component=logs logs_config=integrations msg="could not determine if log file still exists while cleaning positions file" error="stat /root/.pm2/logs/server-out-0.log: permission denied"

See my post on issue 429 for more configuration details

[tonyswumac]

You could try to edit the positions file and remove the line for the old file. Or, you could try to touch a file at the old location (assuming it doesn’t exist anymore) and let it clean itself up.

May be a bug or feature request as well, not sure if this was intentional or not.

[spencerdcarlson1]

That worked for me thanks.

systemctl stop grafana-agent.service
sudo su -l grafana-agent -s /bin/bash
vim /tmp/positions.yaml
# remove lines to unused log files:
#   /root/.pm2/logs/server-error-0.log: "10901"
#   /root/.pm2/logs/server-out-0.log: "8404847"
systemctl start grafana-agent.service