Recommendation Engine With Physical Distance Cutoff

Question

I am looking to develop a recommendation engine for local stores to users. There are approximately 1 million stores in the database and around 1 million users. The 1Mx1M matrix for a user-based collaborative filtering can be prohibitive to calculate. I'm trying to come up with shortcuts. The main one that I can envision is the fact that a user will only typically travel approximately 20 miles to visit a store, so comparing a New York user to one in California is typically a wasted computation. However, this would similarly require a distance matrix between users.

Are there any existing solutions for this? Any tutorials or github examples would be appreciated. I could only find examples using distance calculations between all examples.

Note: if possible I would like to use Python.

Answer 1

I agree with Andy that 1M x 1M is not that big, if you use sparse matrices.

But the problem you are describing is the reason User Based Collaborative Filtering (using distance measures between users) is not used in any large commercial system. It gets prohibitive.

An alternative is Item based, although it won't help you here since you still have 1M products. Or more Netflix like approaches - SVD++ and RBM. I would use one of those because they are meant for big data.

Search for SVD++, you will find a lot of examples. There are also prebuilt libraries like graphML or Mahout (although I would suggest Spark based implementations) that you could potentially use.