# What is the standard procedure for evaluating a user-based CF algorithm with a dataset offline?

I have read some papers and other materials about the evaluation of recommender systems (RS). Most of them discuss the various properties of RS (e.g. accuracy, diversity, etc.), and metrics for different tasks (e.g. RMSE, precision, recall, etc.), and some protocols. But I am still not very clear about data partitioning and the detailed validation procedure. [Shani, Guy, and Asela Gunawardana. "Evaluating recommendation systems." Recommender systems handbook. Springer US, 2011. 257-297.]

For example, I want to evaluate a basic user-based CF algorithm with rating feedback. The data format is user-item-rating. There're 100 users in the dataset. Now I want to use 5-fold cross validation method. This is my way:

1. According to users, I randomly separate the original dataset into 5 folds. Each fold contains 20 users.
2. For all users I set one same time instant, so that before and after this time instant, each user contains a number of ratings. The data of any user x before the instant is for creating user profile, the data after the instant is for validating rating prediction (user x as user being predicted) or just being hidden(user x in neighborhood selection).
3. For each user i in all 5 folds, I create a user profile represented by rating vector{} for user i.
4. For fold-1, for each user i in fold-1 I calculate user similarities between user i and the users in the fold-2 ~ fold-5, and set an appropriate value of k-NN to obtain the neighbourhood for each user.
5. For each user i in fold-1, I get predicted ratings from his neighbourhood users (items no t used by the tested user) by an appropriate rating averaging equation.
6. calculate prediction error for user i.
7. Repeat step 2~6 for each fold (5 times all) and get the average prediction error. This is one round.
8. Use different parameter combinations(e.g. k-NN neighborhood size), repeat step 2~7 with many times, so as to obtain the lowest prediction error (meaning best parameter combination).
9. With the best parameter combination, I get the lowest prediction error, and this is the final evaluation result.

I have two questions:

• Is this procedure correct for evaluating a user-based CF algorithm?
• If so, what are the specific data in this procedure corresponding to the concepts of "training set", "testing set" and "validation set" from Statistics?
• a well formulated question plus showing the amount of research you have already done. Wish more questions would be like that one. Commented Aug 6, 2013 at 9:09
• I really enjoyed your question and the way you asked it.. every line was useful..
– mOna
Commented Nov 19, 2015 at 20:56

Regarding "The three sets"

• training set: The users in the training folds restricted to the ratings before the split date. This data is used to build the model.
• validation set: The users in the validation fold restricted to the ratings after the split date. This data is used to evaluate the model.
• testing set: The users put aside beforehand (currently not happening in your setup), restricted to the ratings after the split date. This is done in the following way:
• Use all data in the training and validation set to build the model (restricted to the ratings before the split date)
• Calculate the neighborhood using the user data in the testing set before the split date and calculate the error using the ratings after the split data (standard procedure)

The important point is, that the testing set is used to evaluate what the generalization error and quality looks like when put into practice (after all optimization has been done). If the quality at this step is strongly inferior to the one obtained in the validation set, overfitting has slipped in.

See also this question: What is the difference between test set and validation set?

Regarding inner vs outer crossvalidation

The inner crossvalidation is used to optimize the so called hyperparameters (http://en.wikipedia.org/wiki/Hyperparameter_optimization) like the neighborhood k. It is done by splitting the training set again. The outer crossvalidation is used to evaluate the generalization power and error of this optimization.

Now one may ask: Why resplitting the training data ? Why not just say that

• the training part of the inner crossvalidation corresponds to the training set
• the validation part of the inner crossvalidation corresponds to the validation set
• the validation part of the outer crossvalidation corresponds to the testing set

One can do this. But given this split without any data put aside beforehand, one must be careful not to e.g. rerun the outer crossvalidation with different settings, because in this case you do not have untainted data to use for the final estimation of the generalization power and error.

Regarding the validation procedure ignoring the testing set and hyperparameter optimization argument

Basically, the validation itself is good and valid ! Splitting the users into folds varies the mixture of interests present in the training data meanwhile splitting every user profile at a certain date takes timely aspects like "new items coming in" or "what is hot a the moment" etc. into account.

Some thoughts based on my own experience ...

Repetition of the (outer) crossvalidation

You may repeat the crossvalidation with different user and rating splittings without varying the hyperparameters of the model to gain a more reliable estimate of the generalization error. Beware of repeating it not to often (I recommend 6-10 based on Kohavi's analysis of crossvalidation), since the more repetitions, the higher the probability that the same type of split occurs again.

Contemplation: Drop the splitting at a date ?

One suggestion: Splitting at a certain date has the disadvantage that the amount of training data and validation data may vary heavily across users, depending on their levels of activity. Distinguishing between these types of users may help to first build a good recommender, then make it work for small amounts of data, too.

To account for this one can ...

• either do not use users for validation where the amount of training data is to small.
• drop the splitting at the date and split the ratings randomly (e.g.2/3 / 1/3 or something like this). This is only appropriate for items with less dynamics (books, movies) then items with a high dynamics (fashion). To somehow balance this, myself has restricted the validation to items which occurred at least once in a certain time frame (e.g. a month). The time frame is chosen based on the dynamics of the items, i.e. it represents the time frame where the item / rating occurrences can be considered as stable.

Contemplation: Drop the splitting of users ?

When the recommender system is applied in practice, all users with activity up to the launch date are known. So splitting the users in such a way that some are put into the training folds and other in the validation folds is not very realistic. It would be more realistic to just split the ratings per user at a certain date or just random (2/3 / 1/3), using all data from all users in the training part of the split for building and the rest for validation accordingly.

HOWEVER, the only difference to the user-and-rating-splitting is, that the rating-splitting-only can utilize the ratings of the user in the validation fold before the date split. So all in all not much of a difference. The user-and-rating-splitting makes the model training a little bit harder, but forces more stability on the other hand. It is some sort of regularization.