How to treat rare / new items in the validation of a recommender system? I'm working on movie recommendation algorithm. The data set consists of about 40 million ratings (user, film, rating). I want to separate the ratings into two groups - training set and probe set. The training set will be used for training, probe set for checking that the algorithm doesn't overfit the data.
My first plan was to try to follow this approach: http://www.netflixprize.com/community/viewtopic.php?id=332. They create the probe set by taking the last 3 ratings from every user (and every user is guaranteed to have 20+ ratings). But the problem is, what if there is some really new film whose all ratings are in the probe set? The training algorithm wouldn't have any info about this film.
 A: A collaborative filtering system is trained only using ratings on user-item pairs. When a user or item (in this case, movie) enters such a system for the first time, the system will not be able to generate a prediction for it. This is known as the cold-start problem. One way to mitigate this problem is by using some other feature-data about the movies, for example the genre, lead actors, etc. if available.
In this case, it doesn't make sense to test the system on a movie for which it doesn't have ratings data. If a rating has to be generated for such movies, you may not be able to do much better than assigning the user-average rating, which is not very informative.
The probe set can be built by taking the last three ratings of each user, and then removing those movies which occur only in the probe set. After validation, the final model can be built using all data points.
See http://en.wikipedia.org/wiki/Cold_start
A: In my opinion you should use this approach (separate the dataset in two groups) only during the evaluation of the algorithms. After you verify that your algorithm is correct you should use the entire dataset as training set.
Do you have a look to my project, reco4j? we have a similar goal.
