Is there one splitting strategy for both K-NN and Matrix Factorization recommender systems?

Question

I am researching several different recommender systems, some of which are based on a user K-Nearest Neighbour algorithm and some of which are based on a matrix factorization algorithm. My dataset is sparse and consists of interactions between users and items. Now, I am trying to split the data into a train and test set. My initial idea would be naively splitting the known interactions randomly (80-20). However, this is impossible, as KNN and MF algorithms have different requirements:

• KNN: The test set should ideally contain the full profile of the user, i.e. it should contain all the known interactions of a set of test users.
• MF: The test set should only contain users who are also in the training set, i.e. at least one interaction of every user should be present in the training set.

The two requirements clearly conflict. However, for standardization purposes, there should be one dataset to cover both types of algorithms. I cannot find much information about this in the literature. Is there a standard way of splitting that can be used for both scenarios?

Answer 1

I'm not sure where did you found the requirements and it lacks context, but I don't fully agree with them. First, notice that "all the interactions" and "at least one interaction" are not in conflict, but the second one is just more liberal.

Second, recall how the algorithms work. In $k$-NN, you make the recommendation based on the smallest distance among neighbors. In matrix factorization, you predict the missing ratings of the matrix. In both cases, you need some historical data where you observe the interactions of the users with items so that you can make recommendations for the items that were not rated by the users yet. To test it, you need relevant data with users who rated some historical items (but there are ways to deal with cold start) and the new items to be scored. In $k$-NN it will be a bigger technical problem to have enough historical ratings so you can calculate meaningful distance, but in both cases, your test set should be rather about what is the actual data that you would see in a production environment. If in a production environment you have many cases where users rated only a single item and you need to make a recommendation based on this data, this is a hard constraint on how would you approach the problem, not only validate the model. You also want the test set to resemble reality so that you know that the results are meaningful. If you would need to subset the data only to validate the model, then it is a likely hypothesis that you could use the model only for this subset of users as well, so how would you make the recommendations for the rest of the users?