I am developing a content-recommender Python system and most of my items (~8 millions) are static so I have thought about pre-computing the top 150 similar items for each item. This way, when a user selects any of them, I will already know which ones should I recommend him.

I have found this scikit function sklearn.metrics.pairwise.euclidean_distances. The problem I have is that it gives back the redundant form of the distance matrix. This is a 8Mx8M matrix. At the end I just need a 8Mx150 distance matrix. So, is there any possible way of getting a distance matrix in a more condensed version directly?

I mean directly because I know that I can remove the redundant info later. However I don't think its a good idea to have a step in the process wherein a 8Mx8M matrix resides in memory. I have also thought about measuring the top 150 distances item by item, but I have the feeling that this will be slower than using already implemented functions in Numpy, SciPy, scikit-learn etc.


After looking at both the scikit-learn euclidean_distances and at the scipy.spatial.distance functions I couldn't find a way to get only the top 150 most similar items out of the box.

I can see three options:

  1. You tweak the code in one of these libraries to fit your needs (Probably this will lead you to the most efficient solution);
  2. You measure the 150 smallest distances item by item (Might not be efficient, but it's not hard to implement);
  3. You split your 8M items into random sub samples and calculate the similarities only inside each sub sample (This won't lead you to the most similar 150 items but might be a good approximation).

Of these I would go with option 2 as, although it might be inefficient, you said you only have to run it once.

  • 1
    $\begingroup$ You have given me an idea: What about 2 and 3 together?: I'll have a <chunk size>x8M matrix each time and I will get the top 150 items. Of course I will have to find a reasonable <chunk size> that fit my computer. $\endgroup$ Mar 30 '16 at 19:20
  • $\begingroup$ Yes I think that should work, still a <chunk size>x8M matrix it's not easy to fit in memory. $\endgroup$ Mar 31 '16 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.