How to divide dataset into training and test set in Recommender Systems? I am working on two simple recommender systems - Collaborative (item-item a user-user) and Content Based. I would like to evaluate prediction accuracy of these systems.
I am used to divide dataset into two parts - training set and evaluation dataset. Is it needed to do in Recommender systems? I have no idea how to do it, because there is no learned "model" and predictions computation is based on similarities.
Any thoughts?
 A: Typically, it is the events that are split, either based on time or on the users. That is, given a set of users $U$, you have a set of events $E = \{ (t,u,i) \}$ that the user $u$ viewed/rated/bought item $i$ at time $t$. On option is to pick $t_1$ and $t_2$ such that $t_1<t_2$ and define $E_{\mathrm{training}} = \{ (t,u,i) \in E, t < t_1\}$, $E_{\mathrm{test}} = \{ (t,u,i), t > t_2 \}$ (a gap $t_2-t_1>0$ is often used to elminate leakage from the training set to the test set, e.g. ongoing sessions). Another possibility is to take a disjoint partition $U_1,U_2$ of $U$ and let $E_{\mathrm{training}} = \{ (t,u,i) \in E, u \in U_1\}$, $E_{\mathrm{test}} = \{ (t,u,i), u \in U_2 \}$ (the tacit assumption being that the behaviour of different users is independent).
A: Assuming users' interest profile is static, you can do a normal cross-folding process for evaluation, partitioning over the rating logs.
If you assume dynamic user interest, then your model needs to be re-evaluated every now and then to capture the changes (which is what real world systems do). Therefore within a time-block, you are still assuming things go static. So you can perhaps take all ratings within a month and perform normal cross-folding.
If you are going to cross-fold, please try to do it with number of folds around 5 - 10.
Or take a holdout set like @sandris suggested.
Good read: http://www.contentwise.tv/files/An-evaluation-Methodology-for-Collaborative-Recommender-Systems.pdf
