Hi Bernhard, welcome to the forum 
This is a common question, and your instincts are correct. There are a couple of features in k6 that can help you with this: SharedArray, which will minimise the memory needed to load the data, and the k6/execution module, which will give you a counter to help you pick unique records from the data. See this post for an example, and the rest of the thread for other, now deprecated, examples.
Have a search on the forum as well, there are other useful threads around this topic, though I think the above one is the most relevant.
Good luck!