The logs of loki are flooded with “error deleting chunk” and “failed to delete key”. What can I do?
Here is a short excerpt:
2021-10-06 10:49:11
level=warn ts=2021-10-06T08:49:11.490376481Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0007" value=fake/db0094787c102fb2:17b32c252e4:17b32ffaa21:90488e4 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyYzI1MmU0OjE3YjMyZmZhYTIxOjkwNDg4ZTQ=: no such file or directory"
2021-10-06 10:49:11
level=error ts=2021-10-06T08:49:11.490366221Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/db0094787c102fb2:17b32c252e4:17b32ffaa21:90488e4 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyYzI1MmU0OjE3YjMyZmZhYTIxOjkwNDg4ZTQ=: no such file or directory"
2021-10-06 10:49:11
level=warn ts=2021-10-06T08:49:11.490348506Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006" value=fake/db0094787c102fb2:17b325ade68:17b3291ccf9:a0b80045 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyNWFkZTY4OjE3YjMyOTFjY2Y5OmEwYjgwMDQ1: no such file or directory"
2021-10-06 10:49:11
level=error ts=2021-10-06T08:49:11.490337836Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/db0094787c102fb2:17b325ade68:17b3291ccf9:a0b80045 err="remove /var/lib/loki/chunks/ZmFrZS9kYjAwOTQ3ODdjMTAyZmIyOjE3YjMyNWFkZTY4OjE3YjMyOTFjY2Y5OmEwYjgwMDQ1: no such file or directory"
2021-10-06 10:49:11
level=warn ts=2021-10-06T08:49:11.490314338Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u001c" value=fake/5ef3d04f7676711b:17b32e535bf:17b331483ac:e5e92ce8 err="remove /var/lib/loki/chunks/ZmFrZS81ZWYzZDA0Zjc2NzY3MTFiOjE3YjMyZTUzNWJmOjE3YjMzMTQ4M2FjOmU1ZTkyY2U4: no such file or directory"
2021-10-06 10:49:11
level=error ts=2021-10-06T08:49:11.490292925Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/5ef3d04f7676711b:17b32e535bf:17b331483ac:e5e92ce8 err="remove /var/lib/loki/chunks/ZmFrZS81ZWYzZDA0Zjc2NzY3MTFiOjE3YjMyZTUzNWJmOjE3YjMzMTQ4M2FjOmU1ZTkyY2U4: no such file or directory"
2021-10-06 10:49:11
level=warn ts=2021-10-06T08:49:11.490174581Z caller=marker.go:300 msg="failed to delete key" key="\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0001" value=fake/80b3523f195c331d:17b32e535d3:17b330fdcb1:68a3bbde err="remove /var/lib/loki/chunks/ZmFrZS84MGIzNTIzZjE5NWMzMzFkOjE3YjMyZTUzNWQzOjE3YjMzMGZkY2IxOjY4YTNiYmRl: no such file or directory"
2021-10-06 10:49:11
level=error ts=2021-10-06T08:49:11.490134295Z caller=retention.go:219 msg="error deleting chunk" chunkID=fake/80b3523f195c331d:17b32e535d3:17b330fdcb1:68a3bbde err="remove /var/lib/loki/chunks/ZmFrZS84MGIzNTIzZjE5NWMzMzFkOjE3YjMyZTUzNWQzOjE3YjMzMGZkY2IxOjY4YTNiYmRl: no such file or directory"
2021-10-06 10:49:11
level=info ts=2021-10-06T08:49:11.489476328Z caller=table_manager.go:324 msg="synching tables" expected_tables=15