Ability to merge two loki databases into a single database

Grafana Loki
1

I have been upgrading our logging server to a newer platform over the past several months. In the process of doing this upgrade I now have two instances of loki running, one with legacy logs, and one with the current logs. I plan to shut off the legacy instance of loki and only use the current instance. I would like to migrate the log data from the legacy server to my current server and merge it to the loki database stored there. Is this possible? If so, how do I do that?

2

What’s different between your old cluster and new cluster?

Without knowing what the difference is, the safest way is probably do an export/import. You should be able to write a script that reads from old script and writes into new cluster, couple of things to note of course:

  1. You’ll want to pay close attention to limits. If you try to grab too much log at once you might hit a limit.

  2. You can’t write old logs to a log stream if there are newer logs present already (even with the option to reject old log disabled). So if you are using the same labels, you will most likely have to change the labels from your old cluster (for example set another label that says old_cluster: yes or something like that.

3

The clusters are exactly the same. It sounds like trying to move my old logs to the new cluster won’t work straight up. Can I export/import the new cluster to the old cluster, delete the new cluster and the move the old cluster back again.

4

If the two clusters are identical, you may be able to move the index files and chunks (maintain the directory structure of course). But I don’t know how you safely test that.

If you don’t have a lot of data on the new cluster then maybe retaining the old cluster would be easier.

5

I can’t see any documentation or examples on how to import/export logs from loki. I’m a bit of a neophyte with loki. It is set up on our OpenShift server and I use a spring boot connection to write my logs to Loki. Any ideas?

6

As far as I know there isn’t a built-in import/export functionality. Fortunately the APIs are pretty easy to use, and we just write our own import / export scripts with Python.

7

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.