My dataset consists of binary preferences ($0$ or $1$) given by users on items like this:
User-ID | Item-ID | Preference
If a user has not given a preference to an item, then it is not in the dataset. (In essence this means there are three possible values in a user-item relation: $0$, $1$ or unknown.)
I want to test some Collaborative Filtering algorithms on my dataset, but I want to do this with stratified cross-validation. I know the general idea on cross-validation and how it works, but I having a little trouble understanding it in the context of Collaborative Filtering.
Correct me if I'm wrong, but stratified cross-validation means that we roughly have an equal percentage of each target class in each fold. In other words, in every fold there has to be the same amount of preferences with value $1$ and the same amount of preferences with value $0$.
If my above reasoning is correct, does this hold the same way when applying this in a collaborative filtering setting? I'm having a little trouble understanding this as with the above method we would have in every fold preferences with $1$ and preferences with $0$, but it is still possible to have folds where users are not presented in, affecting the results of the evaluation.
Question: When applying stratified K-fold cross-validation in a collaborative filtering setting, do we only look at a roughly equally distribution of $1$'s and $0$'s in every fold? Or do we also want a roughly equally distribution of users (or items) across folds?