Regarding "The three sets"
- training set: The users in the training folds restricted to the ratings before the split date. This data is used to build the model.
- validation set: The users in the validation fold restricted to the ratings after the split date. This data is used to evaluate the model.
- testing set: The users put aside beforehand (currently not happening in your setup), restricted to the ratings after the split date. This is done in the following way:
- Use all data in the training and validation set to build the model (restricted to the ratings before the split date)
- Calculate the neighborhood using the user data in the testing set before the split date and calculate the error using the ratings after the split data (standard procedure)
The important point is, that the testing set is used to evaluate
what the generalization error and quality looks like when put into practice (after all optimization has been done). If the quality at this step is
strongly inferior to the one obtained in the validation set, overfitting has slipped in.
See also this question: What is the difference between test set and validation set?
Regarding inner vs outer crossvalidation
The inner crossvalidation is used to optimize the so called hyperparameters (http://en.wikipedia.org/wiki/Hyperparameter_optimization) like the neighborhood k.
It is done by splitting the training set again. The outer crossvalidation is used to evaluate the generalization power and error of this optimization.
See also this question: Nested cross validation for model selection
Now one may ask: Why resplitting the training data ? Why not just say that
- the training part of the inner crossvalidation corresponds to the training set
- the validation part of the inner crossvalidation corresponds to the validation set
- the validation part of the outer crossvalidation corresponds to the testing set
One can do this. But given this split without any data put aside beforehand, one must be careful not to e.g. rerun the outer crossvalidation with different settings,
because in this case you do not have untainted data to use for the final estimation of the generalization power and error.
Regarding the validation procedure ignoring the testing set and hyperparameter optimization argument
Basically, the validation itself is good and valid ! Splitting the users into folds varies
the mixture of interests present in the training data meanwhile splitting every user profile
at a certain date takes timely aspects like "new items coming in" or "what is hot a the moment" etc.
Some thoughts based on my own experience ...
Repetition of the (outer) crossvalidation
You may repeat the crossvalidation with different user and rating splittings without varying the hyperparameters
of the model to gain a more reliable estimate of the generalization error. Beware of repeating it not to often
(I recommend 6-10 based on Kohavi's analysis of crossvalidation), since the more repetitions, the higher the probability
that the same type of split occurs again.
Contemplation: Drop the splitting at a date ?
One suggestion: Splitting at a certain date has the disadvantage that the amount of training data and validation data may vary heavily across users, depending on their levels of activity. Distinguishing between these types of users may help to first build a good recommender, then make it work for small amounts of data, too.
To account for this one can ...
- either do not use users for validation where the amount of training data is to small.
- drop the splitting at the date and split the ratings randomly (e.g.2/3 / 1/3 or something like this). This is only appropriate for items with less dynamics (books, movies) then items with a high dynamics (fashion). To somehow balance this, myself has restricted the validation to items which occurred at least once in a certain time frame (e.g. a month). The time frame is chosen based on the dynamics of the items, i.e. it represents the time frame where the item / rating occurrences can be considered as stable.
Contemplation: Drop the splitting of users ?
When the recommender system is applied in practice, all users with activity up to the launch date are known. So splitting the users in such a way that some are put into the training folds and other in the validation folds is not very realistic. It would be more realistic to just split the ratings per user at a certain date or just random (2/3 / 1/3), using all data from all users in the training part of the split for building and the rest for validation accordingly.
HOWEVER, the only difference to the user-and-rating-splitting is, that the rating-splitting-only can utilize the ratings of the user in the validation fold before the date split. So all in all not much of a difference. The user-and-rating-splitting makes the model training a little bit harder, but forces more stability on the other hand. It is some sort of regularization.