I am looking to develop a recommendation engine for local stores to users. There are approximately 1 million stores in the database and around 1 million users. The 1Mx1M matrix for a user-based collaborative filtering can be prohibitive to calculate. I'm trying to come up with shortcuts. The main one that I can envision is the fact that a user will only typically travel approximately 20 miles to visit a store, so comparing a New York user to one in California is typically a wasted computation. However, this would similarly require a distance matrix between users.

Are there any existing solutions for this? Any tutorials or github examples would be appreciated. I could only find examples using distance calculations between all examples.

Note: if possible I would like to use Python.

  • 1
    $\begingroup$ You're gonna want to learn everything you can about sparse matrices. 1M users by 1M stores will be fine as long as you don't instantiate (or worse, try and invert) the whole thing. $\endgroup$
    – Andy Jones
    Commented Dec 16, 2014 at 22:07

1 Answer 1


I agree with Andy that 1M x 1M is not that big, if you use sparse matrices.

But the problem you are describing is the reason User Based Collaborative Filtering (using distance measures between users) is not used in any large commercial system. It gets prohibitive.

An alternative is Item based, although it won't help you here since you still have 1M products. Or more Netflix like approaches - SVD++ and RBM. I would use one of those because they are meant for big data.

Search for SVD++, you will find a lot of examples. There are also prebuilt libraries like graphML or Mahout (although I would suggest Spark based implementations) that you could potentially use.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.