2
$\begingroup$

I am working on two simple recommender systems - Collaborative (item-item a user-user) and Content Based. I would like to evaluate prediction accuracy of these systems.

I am used to divide dataset into two parts - training set and evaluation dataset. Is it needed to do in Recommender systems? I have no idea how to do it, because there is no learned "model" and predictions computation is based on similarities.

Any thoughts?

$\endgroup$
2
$\begingroup$

Typically, it is the events that are split, either based on time or on the users. That is, given a set of users $U$, you have a set of events $E = \{ (t,u,i) \}$ that the user $u$ viewed/rated/bought item $i$ at time $t$. On option is to pick $t_1$ and $t_2$ such that $t_1<t_2$ and define $E_{\mathrm{training}} = \{ (t,u,i) \in E, t < t_1\}$, $E_{\mathrm{test}} = \{ (t,u,i), t > t_2 \}$ (a gap $t_2-t_1>0$ is often used to elminate leakage from the training set to the test set, e.g. ongoing sessions). Another possibility is to take a disjoint partition $U_1,U_2$ of $U$ and let $E_{\mathrm{training}} = \{ (t,u,i) \in E, u \in U_1\}$, $E_{\mathrm{test}} = \{ (t,u,i), u \in U_2 \}$ (the tacit assumption being that the behaviour of different users is independent).

$\endgroup$
0
$\begingroup$

Assuming users' interest profile is static, you can do a normal cross-folding process for evaluation, partitioning over the rating logs.

If you assume dynamic user interest, then your model needs to be re-evaluated every now and then to capture the changes (which is what real world systems do). Therefore within a time-block, you are still assuming things go static. So you can perhaps take all ratings within a month and perform normal cross-folding.

If you are going to cross-fold, please try to do it with number of folds around 5 - 10.

Or take a holdout set like @sandris suggested.

Good read: http://www.contentwise.tv/files/An-evaluation-Methodology-for-Collaborative-Recommender-Systems.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.