My dataset consists of binary preferences ($0$ or $1$) given by users on items like this:

User-ID | Item-ID | Preference

If a user has not given a preference to an item, then it is not in the dataset. (In essence this means there are three possible values in a user-item relation: $0$, $1$ or unknown.)

I want to test some Collaborative Filtering algorithms on my dataset, but I want to do this with stratified cross-validation. I know the general idea on cross-validation and how it works, but I having a little trouble understanding it in the context of Collaborative Filtering.

Correct me if I'm wrong, but stratified cross-validation means that we roughly have an equal percentage of each target class in each fold. In other words, in every fold there has to be the same amount of preferences with value $1$ and the same amount of preferences with value $0$.

If my above reasoning is correct, does this hold the same way when applying this in a collaborative filtering setting? I'm having a little trouble understanding this as with the above method we would have in every fold preferences with $1$ and preferences with $0$, but it is still possible to have folds where users are not presented in, affecting the results of the evaluation.

Question: When applying stratified K-fold cross-validation in a collaborative filtering setting, do we only look at a roughly equally distribution of $1$'s and $0$'s in every fold? Or do we also want a roughly equally distribution of users (or items) across folds?

  • $\begingroup$ What did you end up doing? What were the results? What would you recommend? Was your method scalable? $\endgroup$ – Adorn Feb 20 '18 at 2:41
  • $\begingroup$ @Adorn, sorry for the late reply. I don't remember the exact details, but we ended up saying that stratification in this context means having roughly the same distribution of items of each user in each fold. In order to achieve this though we did need to put a threshold of a minimum of items per user (and minimum of users per item). The results worked out well, and while the approach seemed to work, I do recommend looking at it again from a more theoretical point of view to see if it's the right way. As for scalable, we had a data set with up to 1 million ratings and that worked fine. $\endgroup$ – Floris Devriendt Mar 28 '18 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.