I am developing a content-recommender Python system and most of my items (~8 millions) are static so I have thought about pre-computing the top 150 similar items for each item. This way, when a user selects any of them, I will already know which ones should I recommend him.
I have found this scikit function sklearn.metrics.pairwise.euclidean_distances. The problem I have is that it gives back the redundant form of the distance matrix. This is a 8Mx8M matrix. At the end I just need a 8Mx150 distance matrix. So, is there any possible way of getting a distance matrix in a more condensed version directly?
I mean directly because I know that I can remove the redundant info later. However I don't think its a good idea to have a step in the process wherein a 8Mx8M matrix resides in memory. I have also thought about measuring the top 150 distances item by item, but I have the feeling that this will be slower than using already implemented functions in Numpy, SciPy, scikit-learn etc.